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Executive  Summary 

This  paper  describes  the  peer  reviewed  use  of  multi-attribute  decision  analysis  in  the  Anny’s 
Capabilities  Needs  Analysis  (CNA).  The  CNA’s  purpose  is  to  enable  prioritization  of  Army  Capabilities 
Development  and  Resourcing  to  meet  Joint  Warfighting  needs  through  a  capabilities-based  analysis  of 
Anny  Warfighting  Challenges  across  Anny  Capabilities  Development  Imperatives  (i.e.,  DOTMLPF-P) 
given  Strategic  Guidance  and  results  from  other  key  capabilities  development  work.  Director,  ARCIC’s 
intent  for  CNA  completion  was  to  provide:  prioritized  DOTMLPF-P  fielded  and  programmed  solutions 
alternatives  across  Anny  Warfighting  Challenges  to  sustain  in  the  Program  Objective  Memorandum 
development  that  enable  Anny  resourcing  decision  making;  prioritized  capability  gap  alternatives  across 
Anny  Warfighting  Challenges  recognized  by  all  stakeholders  that  help  focus  capabilities  development 
activities;  and  prioritized  recommended  Ideas  for  Non-material  and  materiel  (INMA/IMA)  solution 
alternatives  across  the  most  critical  gaps  across  Anny  Warfighting  Challenges  to  drive  future 
Capabilities  Development.  The  prioritizations  of  the  three  sets  of  alternatives  attempt  to  answer  the 
central  objectives  elicited  from  senior  leadership:  What  are  the  most  valued  fielded  and  programmed 
solutions  (DOTMLPF-P  program/solution  alternatives)  in  which  to  sustain  investment  in  order  to  meet 
the  Army’s  Warfighting  Objectives  over  the  near  and  mid-term;  what  capability  shortfalls  or  gaps  (gap 
alternatives)  exist  where  operational  risk  is  unacceptable;  and  what  opportunities  does  the  Army  have 
and  in  what  INMA/IMA  solutions  should  the  Army  invest,  over  time,  in  order  to  avoid  high  risk 
capability  gaps  in  ensure  its  ability  to  meet  its  Joint  Warfighting  Requirements  and  field  the  Army  of 
2025?  The  latter  question  and  value  model  in  the  triplet  of  objectives  is  the  focus  of  many  of  the  Army 
Force  2025  efforts,  especially  recommendations  on  focusing  Science  and  Technology  efforts  to  help 
solve  or  mitigate  capability  gaps.  These  objectives  focused  development  of  the  CNA’s  three 
prioritization  value  models.  Each  of  these  models  consists  of  four  or  five  hierarchical  levels 
decomposed  below  the  objective  that  enables  operationally  relevant,  contextual  understanding  of  the 
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value  of  each  of  the  three  sets  of  alternatives.  These  levels  include:  6  Support  to  Strategic  Analysis 

compliance  scenarios;  more  than  90  operating  force  formations  at  tactical,  operational  and  strategic 

echelons;  the  Army’s  20  Warfighting  Challenges  and  the  formation’s  set  of  Required  Capabilities  with 

more  than  800  of  their  associated  task,  conditions  and  standard  sets  /mission  essential  tasks;  and  the 

more  than  1700  DOTMLPF-P  solution  alternatives  that  provided  a  degree  of  ability  to  achieve  the  tasks, 

or  the  more  than  800  capability  gap  alternatives  where  the  solution  sets  do  not  fully  achieve  the  tasks 

resulting  in  unacceptable  degree  of  operational  risk,  and  the  set  of  recommended  ideas  for  non-material 

solution  or  ideas  for  materiel  solution  alternatives  to  the  critical  capability  gap.  The  implementation  of 

these  value  models  produced  initial  prioritized  of  DOTMLPF-P  fielded  and  programmed  solutions, 

capability  gaps  and  recommended  gap  solution  approaches.  Once  analyst  made  initial  assessments  a 

refinement  process  using  a  modified  Delphi  method  enabled  final  recommendations  that  accounted  for 

experience  and  expertise  not  fully  measured  in  the  value  model  during  the  initial  assessment.  This 

resulted  in  a  consensus  with  the  final  prioritizations  across  all  stakeholders.  Analysis,  especially 

sensitivity  analysis  of  the  attributes  used,  showed  that  generally  the  attributes  and  associated  constructed 

scales  enabled  preference  selection  among  the  alternatives.  However,  preference  was  seen  to  be 

sensitive  to  moderate  change  in  attribute  weighted  values  in  required  capability/task  attributes  level  of 

the  value  model  but  were  viewed  to  have  little  adverse  effect  on  overall  result.  This  provided 

confidence  that  the  value  models  provided  the  preferences  for  the  solution/program  alternatives  that  are 

most  valued  and  are  recommended  for  sustained  investment,  gap  alternatives  with  the  most  unacceptable 

risk  so  they  can  be  solved  soonest,  and  investment  is  focused  toward  INMA/IMA  solution  alternatives 

across  those  critical  capability  gaps.  The  prevalent  use  of  constructed  scales  relying  upon  qualitative 

judgment  required  a  significant  manpower  and  time  resource  commitment  suggesting  the  need  to 

investigate  of  the  use  of  more  precise  perfonnance  metrics  for  attributes  representing  elements  in  the 

value  model.  This  could  remove  some  of  the  resources  required  to  detennine  the  alternatives’  relative 
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values  but  the  up-front  cost  to  obtain  the  data  necessary  to  populate  the  value  model  is  likely  to  be 


significant.  Refinements  of  attribute  criterion  measures  are  also  recommended  to  improve  confidence  in 
the  preferences  provided.  Future  improvement  of  the  CNA  value  models  is  planned  to  address  each  of 
these  refinement  recommendations. 
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Introduction 

This  paper  describes  the  use  of  multi-attribute  decision  analysis  in  the  Army’s  Capabilities  Needs 
Analysis  (CNA).  The  CNA’s  two  primary  purposes  require  trade-offs  and  prioritization  of  many 
alternative  capabilities  in  order  to  provide  requirements  development  recommendations  across  Army 
functions  and  formations,  and  to  inform  Anny  resourcing  and  developments  processes  and  decisions  in 
the  Army’s  Program  Objective  Memorandum  (POM).  The  CNA  uses  the  methodology  for  a 
Capabilities  Based  Assessment  (CBA)  as  the  basis  for  its  method  to  prioritize  among  the  alternatives. 
This  paper  expounds  upon  that  method  including:  the  objective  hierarchy  and  value  model  hierarchies 
used  to  analyze  multiple  conflicting  sub-objectives;  the  development  of  an  additive  value  models  to 
recommend  highest  valued  alternatives  including  selection  of  measures  and  development  of  value 
functions  and  their  scales;  the  use  of  a  modified  Delphi  method  to  refine  the  analysis  results;  and  the 
implementation  of  an  open  architecture  web-based  tool  to  collect  and  analyze  data  and  socialize  results. 
Finally,  the  effectiveness  achieved  during  implementation  is  presented  and  areas  for  improvement  are 
discussed. 

The  CNA’s  purpose  is  to  enable  prioritization  of  Anny  Capabilities  Development  and  Resourcing  to 
meet  Joint  Warfighting  needs  through  a  capabilities-based  assessment  (CBA)  across  DOTMLPF-P  given 
strategic  guidance  and  results  from  other  key  capabilities  development  work.  Figure  1  graphically 
depicts  the  CNA  purpose.  Key  elements  of  the  purpose  that  needed  to  be  considered  during  the  analysis 
include:  Identifying  and  assessing  ‘ what  the  Army  must  perform?  ’  in  a  given  scenario;  assessing  a 
solution’s  ability  to  satisfy  what  the  Anny  must  do.  (‘  What  is  programmed,  thus  should  be  sustained?'); 
Identifying  and  assessing  any  resultant  capability  gaps  for  areas  not  satisfied  (‘  What  the  Army  Cannot 
do?')',  Identifying  and  assessing  non-materiel  and  materiel  solutions  (and  interim  solutions)  to  solve  or 
mitigate  the  critical  capability  gaps.  (‘  Where  the  Army  Should  Focus  Future  Investment?  ’). 
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FIGURE  1.  CNA  Purpose 


The  CNA  derives  its  authority  to  achieve  this  purpose  from  the  Army  Title  10  role  to  man,  equip  and 
train  the  land  forces  to  conduct  war.  Additionally,  TRADOC  and  ARCICs  mission  is  key  to  making 
recommendations  on  the  Army’s  Title  10  role  as  laid  out  in  Army  General  Order  #4  (GO  #4),  Army 
Regulation  71-9  (AR  71-9),  Training  and  Doctrine  Command  Regulation  71-20  (TR  71-20)  and  further 
defined  in  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170.011  (CJCSI  3170.011).  The  Director, 
ARCIC’s  intent  for  CNA  completion  was  to  provide  a  prioritized  DOTMLPF  fielded  and  programmed 
solutions  to  sustain  in  the  POM,  and  prioritized  capability  gaps  and  prioritized  recommended  solution 
strategies  for  the  most  critical  gaps  to  inform  future  Capabilities  Development  and  infonn  Anny 
resourcing.  The  goal  for  the  CNA  was  to  assess  both  formations  and  functions  during  the  analysis  in  a 
manner  that  enables  the  prioritization  under  the  methodological  construct  of  a  Capabilities  Based 
Assessment  (CBA). 

The  scope  of  the  analysis  was  established  based  upon  the  intent  and  the  constraints  set  by  ARCIC 
leadership.  The  analysis  covers  nearly  30  proponent’s  responsibility  for  force  development  of  more  than 
50  formations  types  that  conduct  unified  land  operations  represented  in  6  warfighting  functional  areas 
and  3  special  functional  areas,  across  a  widely  diverse  operating  environment  established  in  6  scenarios 
over  a  timeframe  from  today  through  2040.  Additionally,  limitations  on  time  and  manpower  resources 
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resulted  in  TRADOC  Centers  of  Excellence  and  Force  Modernization  proponents  focusing  the  scope  of 
the  analysis  based  upon  their  senior  leader’s  judgment  of  critical  areas  to  assess  resulting  in  an  analysis 
that  sufficiently  addresses  their  area  of  responsibility  vice  comprehensively  analyzing  their  entire  areas 
of  responsibility. 

Analytically,  the  CNA  purpose  and  leadership  intent  can  be  categorized  as  a  multi-attribute  decision 
problem  to  answer  the  set  of  central  questions  since  the  problem  posed  can  be  summarized  as  deciding 
among  various  alternatives  where  those  alternatives  are  valued  differently  across  selected  criteria  (which 
are  derived  from  the  Capabilities  Based  Assessment  (CBA)  methodology).  Drawing  from  the  Decision 
Analysis  branch  of  the  science  of  Operations  Research,  Edwards  and  Barron1  as  well  as  others 
(Kirkwood,  Keeney,  Parnell,  etc.)  suggests  value  modeling  as  a  widely  accepted  technique  in  order  to 
address  the  kind  of  trade-offs  that  are  required  to  detennine  the  ordered  preference  of  alternatives  as 
required  in  the  CNA.  However,  before  pursuing  the  step-by-step  method  suggested  by  Edwards  and 
Barron  as  modified  by  others  to  model  the  multi-attribute  decision  problem  and  produce  preferred 
alternatives  and  ordered  list  of  those  alternative,  a  complete  understanding  of  the  nature  and  extent  of 
the  problem  is  required. 


CNA  Constraints,  Limitations,  Assumptions 

Before  modeling  the  decision  problem  presented  by  the  ARCIC  leadership,  a  detailed  understanding  of 
the  constraints,  limitations  and  assumptions  is  required  to  ensure  error  avoidance  yet  achieve  a 
comprehensive  treatment  of  the  problem.  As  previously  mentioned  and  discussed  here,  the  analysis  is 
constrained  by  the  decision  maker  and  further  limitations  in  the  organizations  ability  to  conduct  the 


1  Edwards,  W.,  Barron,  F.H.,  SMARTS  and  SMARTER:  Improved  Simple  Methods  for  Multiattribute  Utility  Measurement, 
1994,  Organizational  Behavior  and  Human  Decision  Processes,  60,  pp.  306-325. 
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analysis  resulted  in  a  reduction  in  the  possible  scope  of  the  analysis  conducted.  Additionally, 
assumptions  were  required  in  areas  where  data  was  incomplete/unavailable. 

Constraints.  There  were  four  constraints  on  the  implementation  of  the  CNA  and  its  value  models 
imposed  by  leadership  (here  after  referred  to  as  Decision  Makers  (DM)).  First,  use  the  available 
expertise  at  Centers  of  Excellence  (CoE)  to  conduct  analysis,  finishing  the  analysis  annually.  Minimize 
the  extent  of  manpower  and  time  resources  to  achieve  a  “good  enough”  analysis.  Although  practical, 
implementation  caused  considerable  friction  among  participants  due  to  the  vague  nature  of  the 
constraint.  The  implication  of  this  to  the  analysis  is  that  not  all  centers  or  proponent  have  available  the 
most  qualified  experts  to  assess  the  alternatives  (solutions,  fonnations,  gaps,  etc.)  against  the  attributes 
(achievement,  risk,  etc.)  reducing  the  quality  of  the  assessment  of  the  alternatives  against  the  attributes. 
Most  experts  lacked  recent  experience  in  the  fonnations  assessed  and  many  had  been  removed  from 
such  experience  over  several  generations  of  fonnation  design  and  were  not  intimately  familiar  with  the 
most  recent  doctrinal  employment  of  the  fonnation  and  fielded  or  programmed  solution  sets  comprising 
the  fonnation’ s  equipment.  Second,  use  only  the  6  Support  to  Strategic  Analysis  Scenarios2  for 
Capabilities  Development  (set  from  2018  to  2040).  This  constrains  the  operational  environments  (terrain 
and  enemy)  considered  during  the  analysis.  Although  the  scenario  set  was  optimized  to  cover  the  range 
of  military  operations,  most  of  the  detail  provided  for  these  scenarios  focused  primarily  on  Phase  III 
(Dominate)  of  the  Joint  Operations  Process.  This  constraint  could  cause  too  narrow  a  focus  for 
capabilities  development  for  the  Anny  resulting  in  a  ‘sub-optimal’  set  of  recommendations.  This  was 
especially  true  for  Training  and  Engagement  special  functional  areas  where  primary  tasks  are  performed 
in  Joint  Operational  phases  I  and  II.  The  scenarios  were  focused  primarily  at  the  tactical  level  of  war 
with  allowance  for  including  strategic  and  operational  level  where  needed  to  cover  critical  Required 

2Krondak,  et.  at,  TRADOC  Scenario  Gist  Book,  TRAC-F-TR-13-016,  ATTN:  ATRC-FD,  TRADOC  Analysis  Center,  255  Sedgwick 
Ave.,  Fort  Leavenworth,  KS  66027-2345 
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Capabilities  and  Tasks.  These  scenarios  provided  the  conditions  and  context  needed  to  make  value 
judgments  required  during  the  analysis.  Third,  use  only  the  Army  Operating  Concept,  WFF  &  Other 
Functional  Concepts  (representing  how  to  fight  though  2040).  These  concepts  provided  the  critical 
required  capabilities  which  the  analysis  used  as  the  foundation  for  development  of  detailed  tasks, 
conditions  and  standards  to  establish  what  fonnations  would  do  in  each  of  the  scenarios.  Those  concepts 
included  the  Anny  Operating  Concept,  6  Warfighting  Function  (WFF)  Concepts,  the  Training,  Learning 
and  Education  Concept,  the  Engagement  Concept  and  the  Human  Dimension  Concept.  The  Required 
Capabilities  are  hierarchical  in  nature  with  20  Army  Warfighting  Challenges  representing  the 
overarching  enduring  problems  to  be  solved  and  decomposing  into  Army  Warfighting  Functional 
concepts  consisting  of  more  than  188  identified  across  all  the  concepts.  Tasks  were  developed  using 
Universal  Joint  Task  List,  Anny  Universal  Task  List  (ATP  1-03),  and  HQDA  Standardized  Mission 
Essential  Task  Lists  as  a  basis  in  order  to  provide  sufficient  detail  to  enable  assessment  of  the  formation 
or  functional  capabilities.  These  tasks’  standards  were  adjusted,  where  appropriate,  to  account  of  the 
future  operating  environment  detailed  in  the  Army  Concepts.  Forth,  conduct  the  analysis  at  an 
unclassified  level  to  enable  the  broadest  dissemination  of  the  results  and  information.  Where  required  to 
achieve  inclusion  of  critical  areas  that  are  classified,  venues  and  tools  were  provided  to  enable 
consideration  of  classified  critical  areas  and  materiel.  However,  all  releasable  results  are  to  be 
Unclassified,  For  Official  Use  Only. 

Limitations.  Four  limitations  were  noteworthy.  Two  limitations  on  the  breadth  and  depth  of  the  CNA’s 
analytical  rigor  were  necessary  due  to  the  constraints  and  to  avoid  misunderstanding  what  CNA  covers 
or  does  not  cover.  First,  the  analysis  assesses  the  Anny  formations  tasks  and  fielded  and  programmed 
DOTMLPF  solutions  without  consideration  of  dependencies  between  internal  Army  or  Joint  fonnations 
in  order  to  determine  capability  gaps.  This  is  a  critical  limitation  of  the  analysis  since  the  interaction 
possibilities  in  combat  operations  are  nearly  infinite  and  would  be  impossible  to  model  effectively.  The 
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dependency  or  needed  dependence  between  fonnations  was  documented  in  the  gap  solution  assessment. 
Second,  only  the  primary  capability  or  capability  set  provided  by  the  DOTMLPF-P  fielded  or 
programmed  solution  is  assessed  to  avoid  second  and  third  order  effects  and  a  violation  of  mutual 
independence.  Many  solutions  provide  abilities  beyond  their  intended  design  that  provide  unintended 
but  potentially  beneficial  redundancy.  To  avoid  this,  assessors  were  instructed  to  strictly  use  the 
intended  ability  of  a  solution  to  measure  achievement  of  tasks.  A  third  limitation  was  related  to  the 
baseline  for  each  fonnation.  Each  formation  was  assessed  based  upon  its  approved  Table  of 
Organization  and  Equipment  for  Army  2020.  This  enabled  analysis  of  the  effects  of  programmed 
capabilities  introduction  into  the  formations  and  provided  consistency  for  across  an  ever  changing  Army 
force  structure.  The  forth  limitation  involved  assessor  capabilities.  Assessor  learning  curve  associated 
with  newly  developed  tools  and  techniques  caused  assessors  to  focus  away  from  providing  expert 
judgment  during  assessments  and  more  on  tool  functionality.  An  attempt  to  overcome  this  limitation’s 
effect  on  the  analysis  results  was  made  through  multiple  levels  of  review  of  answers  provided  in 
formulation  of  the  results. 

Assumptions:  Five  assumptions  were  required  where  there  were  deficiencies  in  data  completeness  or 
where  the  lack  of  the  assumption  would  not  allow  assessors  to  complete  the  analysis.  First,  everything 

the  Army  must  do  is  performed  by  a  soldier,  government  service  civilian,  or  contractor  in  an  Army 
formation/organization  (Operating  Force  or  Generating  Force).  This  enables  easily  assigning  tasks  to  the 
appropriate  formation  responsible  for  the  task  for  the  Anny.  Second,  the  Anny  will  be  constrained  in 
resources,  thus  decisions  will  be  required  about  what  capabilities  are  needed  and  where  the  resourcing 
should  be  focused.  This  is  the  fundamental  assumption  that  requires  the  development  of  the 
prioritization  models  in  CNA.  If  resources  were  unconstrained,  then  the  Anny  could  have  everything  it 
needs  and  preference  decisions  are  not  required.  Third,  the  Army  invests  resources  to  achieve  the 
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standard  (measure  and  scale)  or  condition  of  a  task  perfonned  by  some  formation/organization  and  those 
investments  are  effectively  managed  to  ensure  fielding  of  the  required  capability.  This  provides  the  link 
between  tasks  and  fielded  and  programmed  solutions  that  require  funding.  That  funding  must  provide 
the  level  of  capability  to  meet  the  task  to  the  standard  under  the  condition,  nothing  more  or  less,  else  the 
funding  is  wasted.  Also,  that  funding  for  the  programmed  solution  must  ensure  that  the  solution  is 
fielded  to  the  Anny  in  order  to  realize  the  provided  capability.  Forth,  the  Anny  (2020)  formations 
assessed  are  organized,  equipped  and  manned  to  100%  fill  of  the  Table  of  Organization  and  Equipment 
for  task  perfonnance  during  assessment  within  the  scenarios.  This  is  essential  to  avoid  nearly  infinite 
possibilities  where  the  Army  knowingly  decided  that  equipment  shortfalls  where  acceptable  in  units  at 
home  station  and  experts  experience  with  the  shortfall  may  introduce  bias  in  the  assessment.  Recent 
history  with  ARFORGEN  and  its  successor  process  verifies  that  this  assumption  is  reasonable.  Fifth, 
the  combination  of  all  included  scenarios  enable  the  assessment  of  all  the  tasks  Army  formations  must 
perform  across  the  six  Joint  Campaign  Phases  (0-V)3  thus  ensuring  that  exhaustive  coverage  of  all 
possible  functions  and  required  capabilities.  Where  this  assumption  was  invalid,  especially  where  the 
scenarios  provided  great  detail  in  one  phase  of  operation  and  only  minimal  details  in  the  others,  the 
scenario  was  expanded  with  the  creation  of  vignettes  within  the  scenario  to  permit  the  assessment  across 
all  functions  and  required  capabilities. 


3  Joint  Publication  5-0,  Joint  Operation  Planning,  Department  of  Defense,  August  11,  2011,  para.  10,  pg  111-41. 
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Analytic  Approach  and  Methodology 


The  CNA  objective,  as  shown  in  Figure  2,  was  established  to  meet  the  ARCIC  Director’s  intent.  It  was 
developed  considering  the  context  of  the  Army  Operating  Concept,  Army  Warfighting  Functional 
Concepts  on  how  to  fight  in  the  future,  and  5  Defense  Planning  scenarios  (EPP,  SSSP,  NEA,  ISC-B,  and 
AFR)  to  enable  prioritization  of  Resourcing  and  Developments  through  a  capability-based  analysis 
across  Warlighting  and  Special  Functions  as  well  as  Formations.  This  objective  was  decomposed  into 
three  issues  and  1 1  Essential  Elements  of  Analysis  (EEA)  to  help  understand  the  extent  of  the  problem 
set  needed  to  answer  the  CNA  objective.  In  essence,  the  CNA  must  determine  preferences  in  three 


areas: 


CNA  Objective 

Friable  prhantiiation  of  Resourcing  and  Developments  through  a  wed  reasoned 
assessment  across  DOTMIPF  AWFC,  Functions  and  Formations  bv 

□  Identifying  and  assessing  what  the  Army  must  do  in  a  given  scenario 

□  Assess  a  program  s  ability  to  achieve  what  we  must  do  What  is  programmed 

□  Identify  and  assess  Capability  Gaps  What  we  cannot  do 

□  Identify  non  materiel  and  materiel  solutions  to  solve  or  mitigate  capability  gaps 
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FIGURE  2.  CNA  Objectives,  Issues  and  Essential  Elements  of  Analysis 


Given  impending  future  Department  of  Defense  resourcing  constraints,  determine  the  ordered  preference 


of  which  of  the  current  programmed  and  fielded  solutions  to  sustain  in  the  force  or  programmed  budget 
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to  ensure  it  meets  its  joint  warfighting  requirements  and  which  can  be  traded-off  with  minimal  risk  to 
pursue  or  buy  a  new  capability  that  mitigates  or  solves  a  high  risk  gap;  determine  which  tasks 
representing  the  Anny’s  Warfighting  Requirements  cannot  be  achieved,  describe  the  capability  gap  that 
results  and  assess  the  operational  risk  of  that  gap  in  the  scenario  set.  This  provides  an  ordered 
preference  for  which  gaps  to  address  soonest;  and  determine  an  ordered  preference  for  the  solutions  that 
solve  or  mitigate  the  gap  to  enable  future  investment  focus. 

These  objectives  were  evaluated  in  the  context  of  the  Keeny’s  nine  desired  properties  of  the  set  of 
fundamental  objectives.4  As  a  result  of  meeting  Keeny’s  properties,  these  objectives  became  the 
fundamental  objectives  as  part  of  the  analytic  approach.  An  approach  (see  Figure  3),  using  a  time- 
phased  application  of  Decision  Analysis  for  multiple  conflicting  objectives5  is  employed  by  developing 
additive  value  models6  for  each  fundamental  objective  since  this  technique  is  suited  to  provide  value 
preferences  between  alternatives  that  can  be  used  to  establish  priorities  and  inform  decisions.  The 
collection  of  data  and  assessment  of  attributes  required  of  the  value  model  is  spread  over  a  period  of  a 
year  in  order  to  mitigate  the  effect  of  constraints  on  manpower.  The  approach  benefits  greatly  from  an 
application  of  web-based  assessment  tools  to  collect  data  from  a  diverse  and  geographically  dispersed 
community  of  practice,  assess  alternatives  and  display  results  in  an  automated  manner.  Finally,  the 
Delphi  method7  is  used  in  achieving  consensus  or  at  least  common  understanding  among  ARCIC  and 
Anny  Senior  leaders  and  stakeholder  about  the  results  of  the  analysis  and  preference  methods. 


4  Keeny,  R.L.,  Value-Focused  Thinking:  A  Path  to  Creative  Decisionmaking,  Harvard  University  Press,  Cambridge, 
Massachusetts,  1992,  p.  82. 

5  Keeny,  R.L.,  Raiffa,  H.,  Decision  With  Multiple  Objectives,  Cambridge  University  Press,  1993. 

6  Parnell,  G.S.,  Driscoll,  P.J.,  Henderson,  D.L.,  Decision  Making  in  Systems  Engineering  and  Management,  2nd  Ed.,  Wiley  and 
Sons,  Inc.,  2011. 

7  Linstone,  H.  A.  and  Turoff,  M.  (eds.,  1975):  The  Delphi  Method  -  Techniques  and  Applications,  Reading:  Addison- Wesley, 
2002. 
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Integrated  Formation  &  Functional  CBA  Methodology 


M  IlM  :<-i  Mni  OP 


FIGURE  3.  CNA  Time-Phased  Analytical  Approach 


Value  Model  Hierarchies 

A  value  model  for  each  of  the  three  fundamental  objectives  has  been  developed,  with  the  assistance  of 
the  TRADOC  Analysis  Center  (TRAC).  These  value  models  account  for  the  logical  decomposition  of 
the  overarching  CNA  objective  into  three  fundamental  objectives,  their  associated  Essential  Elements  of 
Analysis  and  attributes  and  metrics  that  were  later  assessed  during  the  conduct  of  the  analysis  as  shown 
in  Figure  4. 


13 


Unclassified 


ARCIC-TR-015-01 
1  OCT  2015 


FIGURE  4.  CNA  Value  Model  Hierarchies 

The  additive  value  model  of  decision  analysis  theory,  a  widely  accepted  technique,  is  particularly  suited 
to  providing  a  preference  through  value  determination  among  the  three  sets  of  alternatives 
(programmed/fielded  DOTMLPF-P  solutions  to  sustain  in  the  POM,  capability  gaps,  and  DOTmLPF-P 
Ideas  for  Non-Material  and  Ideas  for  Materiel  solutions)  across  the  4  or  5  attributes  derived  from  the 
EEAs.  These  values  can  then  be  used  to  recommend  priorities  between  the  alternatives  using  the 
associated  values. 
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Additive  Value  Model 

The  standard  form  for  an  additive  n-attribute  value  model8  is: 

V(X)  =  V(xir--Xn)  =  (1) 

Where  V(X)  is  the  multi-attribute  values  for  an  alternative  X,  Xi-xn  are  the  n  attribute  scores  for  the 
alternative,  w i  is  the  weight  for  the  ith  attribute,  and  v;  is  the  single  attribute  value  function  for  the  ith 
attribute.  The  weights  are  swing  weights9,  and  are  positive  and  normalized  so  that,  £”=1  wi  —  !•  The 
value  function  measures  the  returns  to  scale  on  the  attributes.  It  is  important  to  understand  that  there  are 
two  conditions  that  must  be  met  in  order  to  appropriately  use  the  Additive  Value  Model:  preferential 
independence  (specifically  mutual  preferential  independence)  and  difference  independence.  Keeney 
and  Raiffa10  provide  a  detailed  explanation  of  these  conditions.  In  summary,  an  additive  value  model  is 
appropriate  when  attributes  are  mutually  preferentially  independent,  that  is,  the  contribution  of  one 
attribute  to  total  value  of  an  alternative  does  not  depend  upon  values  in  other  attributes,  and  an  additive 
value  model  is  appropriate  when  differences  independence  exists,  that  is,  the  specification  of  the 
differences  in  values  within  an  attribute  do  not  depend  on  values  in  other  attributes. 

Adaptation  of  the  Additive  Value  Model  for  use  in  priority  recommendations  in  the  CNA 

Given  the  design  of  the  CNA  hierarchical  value  structure  of  important  attributes  (e.g.,  scenarios  risk  to 
the  Army,  formations  contribution  to  achieve  scenario  objectives,  and  tasks  accomplishment  impact  on 
the  fonnation  mission  accomplishment)  associated  with  the  value  of  the  alternatives  (e.g.,  a  program’s 
ability  to  achieve  the  task,  the  gap’s  risk  and  a  solution  approach’s  ability  to  mitigate  a  gap  and 
feasibility  as  shown  in  Figure  4),  the  use  of  the  additive  value  model  is  appropriate  since  both  mutual 

8  Parnell,  G.S.,  Driscoll,  P.J.,  and  Henderson,  D.L.,  Editors,  Decision  Making  for  Systems  Engineering  and  Management,  2nd 
Editions,  Wiley  Series  in  Systems  Engineering,  Wiley  &  Sons  Inc. ,  2011. 

9  Edwards,  W.,  Barron,  F.H.,  Smarts  and  Smarter:  Improved  Simple  Methods  for  Mulitattibute  Utility  Measurement,  1994, 
Organizational  Behavior  and  Human  Decision  Processes,  60,  pp.  306-325. 

10  Keeny,  R.L.,  Raiffa,  H.,  Decision  With  Multiple  Objectives,  Cambridge  University  Press,  1993,  pp.  104-117. 
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preferential  independence  and  difference  independence  exist.  The  attributes  and  their  returns  to  scales 
were  design  to  insure  mutual  preferential  independence.  For  example,  a  task  attribute’s  contribution  to 
the  total  value  of  a  program/gap/solution  does  not  depend  upon  the  contributions  to  the  total  value  of  a 
program/gap/solution  of  the  other  attributes  (e.g.,  fonnation  or  scenario).  This  is  also  true  about  each  of 
the  attributes.  Also,  the  values  associated  on  the  return  to  scale  for  a  program/gap/solution  approach  is 
not  related  in  any  way  to  the  preference  for  a  scenario,  formation  or  task  where  the  attribute  is  being 
assessed.  Similarly,  the  attributes  were  chosen  and  the  assessment  designed  to  ensure  difference 
independence.  The  selection  or  specification  of  a  value  in  a  programs/gaps/solution  approaches  does  not 
depend  upon  the  selection  or  specification  of  a  value  in  any  of  the  other  attributes.  For  example,  the 
specification  of  solutions  extent  of  achievement  of  a  task  does  not  depend  upon  the  risk  of  scenario 
selection,  contribution  of  fonnation  to  scenario  end-state  accomplishment  or  contribution  of  task  to 
formation  mission  accomplishment.  Thus,  an  adaptation  to  the  multiple  attribute  additive  value  model 
can  be  made.  The  application  of  the  value  model  is  normally  done  by  assessing  local  swing  weights  for 
attributes  at  each  level  of  the  value  hierarchy:  For  obtaining  the  value  of  a  DOTMLPF  fielded  or 
programmed  solution  for  example,  at  the  scenario  level,  weights  Si  are  assessed  for  each  scenario  i  then 
nonnalized  so  that  £?=i  st  —  1;  At  the  fonnation  level,  a  weight  fij  is  assessed  for  each  formation  j  in 
each  scenario  i,  then  normalized  so  that  Yijtifij  —  1  for  all  U  where  a  formation  that  is  not  involved  in  a 
scenario  gets  a  value  of  0.  (e.g.,  /£;-= 0);  At  the  task  level,  a  weight  %  is  assessed  for  each  task  k  in  each 
formation  j  in  each  scenario  i  so  that  Y,k=i  tijk  —  1  for  all 1  scenarios  and  j  formations.  Again,  if  a  task  is 
not  performed  by  a  formation  in  a  scenario  then  t=  0.  Therefore,  wL  —  £_/  Sifijtijk  =  1,  where  w\ 

is  the  additive  model  weight  and  the  local  weights  product  is  Additional  weights  can  be  added 

as  appropriate  for  the  value  model  as  will  be  shown  later  as  long  as  the  Y.f=i  w*  =  1  condition  is  met. 
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DOTMLPF-P  Fielded  and  Programmed  Solution  (Programs)  Alternative  Values 

For  programs  to  sustain  in  the  POM,  the  prioritizing  value  accounts  for  applicability  across  tasks, 
formations,  and  scenarios,  and  the  degree  to  which  the  solution  helps  achieve  those  tasks.  Since  this 
method  is  ratio-based,  comparing  two  items  in  a  final  priority  will  show  both  which  one  is  more 
preferred  and  how  much  more  preferred  it  is  in  comparison  to  another  program.  Thus,  for  a  program  P, 
let  it  have  a  performance  value  xt associated  to  each  task  k,  in  formation  j  and  scenario  i,  where 
vijk(xijk )  is  a  measure  of  ability  to  achieve  the  task  under  the  conditions  to  the  standard  on  the  return  to 
scale  of  the  achievement  value  function,  assumed  to  linear,  as  shown  in  Figure  5. 


Solution  Contribution  to  Task  Achievement 
Return  To  Scale 


Extremely  Large  Extent  -  the  solution,  as 
designed,  has  the  ability  to  achieve  the  entire  task, 
under  all  conditions  and  meets  all  standards. 

Large  Extent  -  the  solution  achieves  most  but  not 
all  parts  of  the  TCS. 

^  Moderate  Extent  -  the  solution  achieves  multiple 
parts  to  the  task,  conditions  or  standards. 

^  Slight  Extent-  the  solution  achieves  less  than  25% 
of  the  task  under  the  conditions  to  the  standard.  At 
least  the  task,  a  condition  and  or  a  standard  can  be 
achieved  with  the  solution. 

Minimal  Extent  -  the  solution  does  not  provide 
successful  achievement  of  the  task  under  any 
condition  to  any  standard. 


Figure  5.  Solution  (Program)  Achievement  of  Task  Constructed  Return  to  Scale  &  Criteria 


Thus,  computation  of  a  program’s  aggregated  value11  across  all  tasks,  formations  and  scenarios,  given 
the  terms  as  defined  above,  can  be  defined  as: 

V(X)=  Zf=lZ”lZfc  =  l5i/i7-tyfc[Vyfc(^yfc)]  (2) 


Bott,  J.M.,  Gross,  A. A.,  Burk,  R.C.,  Parnell,  G.S.,  CAA  Peer  Review  of  ARCICs  CNA  Process,  White  Paper,  Feb  2014. 
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l '  C ' 

Where  the  normalized  scenario  weights  for  each  scenario  i,  are  given  by,  st  —  Js  1 — ,  and  the  attributes 

Z.i=o  Hci 

li,  representing  the  likelihood  that  the  Army  would  be  required  to  conduct  operation  similar  to  those  in 
the  scenario,  and  a  ,  representing  the  consequence  of  Army  mission  failure  in  a  scenario  are  obtained 
using  an  assumed  linear  constructed  return  to  scale  with  well-defined  criteria  based  upon  FM  5-19  as 
shown  in  Figure  6. 


Probability  is  the  "likelihood  that  an  event  will  occur  -  FM5  19, 

PB&  18  and  Glossary  19 

-Unlikely  -The  Army  con  assume  the  condition  or  event  will  not 
occur,  but  It  Is  not  Impossible  Examples  might  include 
detonation  or  containerized  ammunition  during  transport,  river 
flooding  during  a  known  dry  season,  anticipating  a  North  Atlantic 
hurricane  in  May 

■  Seldom  -  Remotely  postibU,  could  occur  at  soma  lima  Usually 
several  things  must  go  wrong  for  It  to  happen  Examples  might 
Include.  »>eat*releted  death,  electrocution 
-Ocrasinrml  -  Occurs  sporadically,  but  Is  not  uncommon 
You  msy  or  may  not  get  through  your  deployment  without  it 
happening  Some  examples  might  Include  unexploded 
ordnance,  fratricide,  anticipating  a  North  Atlantic  hurricane 
in  July 

-Likely  —  Occurs  several  times,  a  common  occurrence.  In 
Illustration,  given  1000  or  so  exposures  without  proper 
controls,  it  will  occur  at  some  point.  Examples  might  Include 
improvised  explosive  devices,  wire  strikes  for  aircraft, 
controlled  flights  Into  tar  ram  end  unintentional  weapons 
discharges 

-Frequent  -  Occurs  very  often,  known  to  happen  regularly 
in  Illustration,  given  500  or  so  exposures  to  the  huzur  d. 
expectthatltwlll  definitely  happen  to  someone  Examples 
of  frequent  occ  ur rente*  «r  *  uehic  !e  rollovers,  r  eer-end 
collision*,  and  heat  injur  y  dm  Ing  a  battalion  physical  training 
run  with  hot  weather  or  non -acclimated  Soldiers, 
anticipating  a  North  Atlantic  hurricane  In  September 


Severity  or  Consequence  is  -expressed  In  terms  of  the  degree  to  which 
an  Incident  will  impact  combat  power,  mission  capability,  or  readiness  " 
ogt  1-9  andUlmiatv  ia 

-Catastrophic 

uforces  faiiureto  achievethe  scenario  objectives  results 
In  an  Ineffective  flohllng  force  where  major  personnel  and 
equipment  losses  preclude  any  additional  action 
aNo  work-  arounds  exist  for  the  force  to  echievethe 
objectives  and  survival  becomes  the  focus 
-Gntical 

©Force's  failure  lo  achieve  the  acenerio  objectives  result* 
in  significant  force  capability  degradation  lossofllfe  and 
destruction  of  equipment 

oFew  if  any.  won-  arounds  to  achieve  the  objectives 
existoi  are  impractical  causing  the  forceto  choose 
unattractive  alternatives  to  prevent  further  loss 
■Marginal 

©Forces  faiiureto  achieve  the  scenario  objectives  results 
in  degraded  Torce  capability  including  some  equipment 
destruction  and  some  personnel  tosses 
of  aw  work  arounds  are  readily  available  to  achieve  (he 
end-stele  end  Implementation  will  lake  significant 
additionaitirn*  and  rasources 
•Negligible 

oForce's  failure  to  achieve  scenario  objective  has  little  or 
no  adverse  impact  on  the  force 

oWoik-  arounds  to  achieve  the  objectives  readily  exist 
and  are  easily  implemented  with  no  major  Impact  on  the 
force 


Figure  6.  Scenario  Operational  Risk  Constructed  Scale  Criteria 


The  normalized  Formation  weights,  f]  ,  are  obtained  for  each  formation  /  in  scenario  i,  using  an 
assumed  linear  constructed  return  to  scale  with  criteria  as  shown  in  Figure  7,  so  that  YjLi  fij  =  1 
within  each  scenario  i. 
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Minimal  Extent  -  the  formation  does  not  contribute  in  any 
measurable  way  to  the  accomplishment  of  mission'objectives 
stated  in  the  scenario 

Slight  Extent  •  the  formation  only  contributes  by  enabling  one 
or  more  other  formations  in  its  contribution  to  the 
accomplishment  of  missiotVobjectives  stated  in  the  scenario 
Moderate  Extent -the  formation  contributes  directly  tofuRy 
accomplishing  at  least  one  of  the  missions/objectives  in  the 
scenario  and  enables  others  in  their  contribution 
Large  Extent  -  the  formation  contributes  directly  to  fully 
accomplishing  multiple  of  the  mission&objectrves  stated  in 
the  scenario  and  enables  the  remaining  formations 
contribution 

Extremely  Large  Extent  -  All  the  missiorVobjeclives  stated  in 
the  scenario  cannot  be  achieved  in  any  manner  without  the 
capabilities  of  the  Formation 


Figure  7.  Formations  Assessed12  and  Formation  Constructed  Scale  Criteria 


The  normalized  task  weights,  %,  for  q  tasks  are  obtained  for  each  formation  j  and  scenario  i  where, 
ti,k  =  qmiik  <h’k —  ,  and  the  impact  on  fonnation  mission  accomplishment  is  ntijk,  and  the  extent  of  force 

^ijk=ilijkaHk 

mission  achievement  given  successful  completion  of  the  task,  a,y*,  are  obtained  through  assessment  of 
constructed  scales  as  shown  in  Figure  8,  so  that  all  tasks  within  a  formation  have  weights  where, 

£k=i  fyk  =  1-  Again,  if  a  task  is  not  performed  in  the  formation  in  the  scenario  it  is  assessed  a  tijk=0. 


Army  Formation  Leader  Book,  DAMO-FM,  Headquarters,  Department  of  the  Army,  v31,  November  21,  2013. 
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Degree  of  Formation  Mission  Accomplishment 

MA  ND  E  C 

Mission  Mission  Essential  to  Critical  to 

Accomplishment  accomplishment  not  Formation/WFF  FormationyWFF 

achieved  without  Task  completely  dependent  Mission  Mission 

on  successful  Accomplishment.  Accomplishment 

performance  of  Task_ n 


Marginal 

Criticality 

Probable 

Criticality 

Critical 

- r4 

Critical 

Marginal 

Criticality 

Probable 

Criticality 

Critical 

Critical 

Not 

Critical 

Probable 

Criticality 

Marginal 

Criticality 

Probable 

Criticality 

Probable 

Criticality 

Not 

Critical 

Not 

Critical 

Marginal 

Criticality 

Marginal 

Criticality 

LOUDE 

the  force  will  not  achieve 
the  objective/end-state  in 
required  time,  resulting  in 
severe  loss  of  life  and 
destruction  of  equipment 

LC/ED 

force  is  likely  to  experience 
widespread  setbacks  in 
achieving  objective, end 
sate,  extended  mission 
execution  times,  limited 
casualties  and  damage  to 

equipment 

SB 

the  force  will  experience 
minor  setbacks  in 
missionlobjective 
execution  time 


Nl 

is  no  adverse  impact  on  the 

force  in  achieving  the 
scenario  mission/obiectives 

if  the  task  is  not  performed 
under  the  condition  to  the 
standard 
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Figure  8.  Task  Constructed  Scale  Criteria 


Thus,  from  this  construct,  a  comparison  of  V(X)  for  each  program  provides  a  means  to  prioritize  the 
programs  according  to  the  preference  value  of  achieving  the  Army’s  tasks  across  formations  and 
scenarios.  Additionally,  this  construct  contributed  to  understanding  and  cogent  decision  making  by 
enabling  the  categorization  of  the  solutions  into  Tiers  of  decreasing  value  to  the  Army  using  the 
operational  logic  shown  in  Figure  9.  It  is  important  to  note  that  the  breaks  between  tiers  were  set  by 
group  of  quintiles  of  solution  value. 
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CNA  Operational  Logic  Construct 


Tierl:  Highest  Risk  to  Mission 
Accomplishment  -Must  Fund/Keep 
Solutions 


Tier  II:  Modest  Risk  to 
Mission  Accomplishment  - 
Fund 

Once  highest  risk  solutions 


Tier  III:  Lowest  Risk  to 
Mission  Accomplishment  - 
Enables  Trade-Off  Analysis 


Solutions  to  Sustain  in  the  POM 
Ordered  By  Preference  Value  arranged  by  Quintile 


Figure  9.  Solution  Tiering  to  Enable  Decision  Making 


Capability  Gap  Operational  Risk  Values 

For  the  Capability  Gap  preference  and  prioritization  to  enable  solution  or  mitigation  of  the  highest  risk 
gap  first,  application  of  the  additive  value  model  in  a  similar  manner  as  described  above  was  developed. 
Again,  the  scenarios,  formations  and  tasks  attributes  remain  the  same,  e.g.,  Sifijtijk.  For  a  Capability 
Gap  G,  let  it  have  a  risk  value  gijkiyijk)  where  ytjk  is  the  gaps  return  on  the  scales  for  the  gap’s 
consequence  and  likelihood  of  occurrence  operational  risk  measure,  as  shown  in  Table  1,  in  scenario  i 
for  formation  j  and  task  A:  in  a  similar  manner  as  solutions. 
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Indicate  the  Gap's  Likelihood  of  Occurrence  by 
Scenario 

Indicate  the  Gap's  Consequence  to  Mission 
Accomplishment 

|  Unlikely  1  Seldom  [Occasional  Likely  |  Frequent  | 

100  o 


100 


T.ikelihood  (l .tV  1  he  freauencv  the  Armv 

Consequence  (c, tV-  Desree  to  which  the  capabibtv  will  imp  act  Armv 

Criteria 

will  encounter  the  capability  gap  in 
operations  represented  by  the  scenario 
set. 

mission  capability,  combat  power,  or  readiness  in  the  operations 
represented  by  the  scenario  set. 

Sub 

Criteria 

F  requently 

The  capability  gap 
occurs  very  o  ften.  known 
to  happ  en  regularly  for 
conditions  in  the 

Op  erations  represented 
in  the  scenazio(s). 

Catastrophic 

The  existence  of  the  gap  results  in  complete  mission 
failure  for  the  Arms  in  the  ooerationFs'i  retire  sente  d  in 
scenario(s).  This  risk  is  inclusive  o f  e a ch  o f  the 
following  indicators  in  the  scenario  set:  loss  of  ability  to 
accomplish  the  mission;  death  or  permanent  total 
disability  of  much  of  the  force;  loss  of  maj  or  or  mission- 
critical  systems  or  equipment;  maj  or  property  or  facility- 
damage;  severe  environmental  damage;  mission-critical 
security  failure;  unacceptable  collateral  damage.  The  gap 
cannot  be  mitigated  by  any  adjustment  of  tactical 
procedures,  techniques  given  the  available  DOTMLPF 
fielded  and  programmed  solutions. 

Likely 

The  capability  gap 
occurs  for  multiple 
conditions,  and  is 
common  occurrence  in 
operations  represented 
by  the  scenano(s). 

C  ritic  a  1 

The  existence  of  the  gap  results  in  severely  degraded 
Armv  mission  c an abilitv  or  readiness  in  on erationf s"> 
represented  in  the  scenario(s).  This  risk  is  inclusive  of 
each  of  the  following  indicators  in  the  scenario  set: 
permanent  partial  disability  or  temporary  total 
disability;  extensive  maj  or  damage  to  equipment  or 
systems;  significant  damage  to  property  or  the 
environment;  security  failure;  significant  collateral 
damage.  Significant  changes  in  tactical  procedures  and 
techniques  and  a  dj  ustment  to  distribution  of  existing 
fielded  and  programme d  DOTMLFP  solutions  required 
to  mitigate  gap  to  acceptable  risk. 

Occasionally 

The  c  ap  ability  gap 
occurs  sporadically 
under  some  of  the 
conditions  m  the 
op  erations  repre sente  d 
by  the  scenano(s),  but  is 
not  uncommon  to  Army- 
operations. 

Ala  rginal 

The  existence  of  the  gap  causes  degraded  mission 
caD abilitv  or  Ann v  readiness.  The  sau  canbe 
mitigate d  by  dedicate  effort  to  change  tactic s. 
t e chmque s  and  redistribute  available  fielded  or 
programmed  DOTMLPF  solutions.  The  risk  is 
inclusive  of  the  following  indicator:  minor  damage  to 
e qmpment  or  systems,  property,  or  the  environment; 
lost  days  due  to  injury  or  illness;  minor  damage  to 
prop ertv  or  the  environment. 

Seldo  m 

The  occurrence  o f  the 
gap  is  remotely  possible; 
it  could  occur  under  only- 
very  few  conditions 
during  the  operations 

scenario(s). 

Negligible 

The  existence  o  f  the  gap  haslittle  or  no  a  dv  erse  imp  a  ct 
on  mission  can  abilitv  of  the  Armv  for  the  scenario's') 
■where  it  occurs.  Gap  is  easily  mitigated  with  tactics, 
techniques  and  procedures  with  the  existing 

DOTMLPF  solution  set  available  to  meet  standards. 

Unlikely 

The  capability  gap  is  not 
likely  to  occur,  but  it  is 
not  impossible  to  occur 
in  the  operations 

sc  enario  fs  j. 

Table  1.  Gap  Risk  Criteria  and  Sub  Criteria  and  Scales 


However,  combinations  of  degrees  likelihood  of  gap  occurrence  (frequent,  likely,  occasional,  seldom, 
unlikely)  and  consequence  of  the  gap  on  the  Army  mission  accomplishment  (catastrophic,  critical, 
marginal,  slight)  in  the  scenario  are  categorized  into  four  levels  of  relative  risk  as  shown  in  Figure  10. 
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Definitions 


Extremely  High  Risk  -  The  Gap  is  will  occur  in  all  operations  (ranging  from  likely 
tofrequent  occurrence)  and  the  Army  has  lost  the  ability  to  accomplisn  the  mission 
as  a  result.  It  is  deemed  that  it  is  not  possible  to  mitigate  the  gap,  given  the  fielded 
and  programmed  DOTMLPF  solution  set  available,  on  the  spot  with  tactical 
procedural  changes  or  addition  of  other  resources  (even  from  outside  the 
operation)  during  performance  of  the  taskin  the  mission  set  represented  by  the 
scenario(s).  This  implies  that  the  risk  associated  with  this  capability  gap  will  have 
severe  consequences  (critical  or  catastrophic)  beyond  those  associated  with  this 
specific  operation  in  the  scenario(s). 

High  Risk-  The  gap  is  likely  to  occur  (ranging  from  seldom  to  likely)  in  one  or 
more  of  the  mission  sets  represented  by  tnescenario(s).  The  Army’s  ability  to 
accomplishthe  mission  is  severely  degraded  requiring  one  or  more  drastic 
adjustments  intactical  procedures  ana  techniques  and  commitment  of  additional 
resources,  given  the  availability  fielded  and  programmed  DOTMLPFsolution  set,  to 
mitigatethegap.  This  implies  that  the  risk  associated  with  this  capability  gap 
causes  consequences  for  the  commander  associated  with  this  other  elements  of 
the  operation  in  the  Scenario. 

-The  gap  occurs  infrequently  (ranging  from  unlikely  to  seldom)  in 
one  or  more  of  the  mission  sets  represented  by  the  scenario(s).  The  Army  can 
accomplishthe  mission  with  the  existing  fielded  and  programmed  DOTMLPF 
solution  set  available  but  requires  adjustment  of  tacticaTprocedures  and 
techniques  and  reallocation  of  existing  resources  to  mitigate  the  gap.  This  implies 
the  risk  associated  with  this  capability  gap  causes  acceptable  consequences 
(marginal  or  critical)  for  the  commander  regarding  other  elements  of  the  operation 
in  the  scenario. 

Low  Risk -Expected  losses  have  little  or  no  impact  on  accomplishing  the  mission. 
Injury,  damage,  or  illness  are  not  expected,  or  may  be  minor  and  have  no  long  term 
impact  or  effect. 


Risk  Assessment  Matrix 


Is 


3 


11’ 


Unlikely 

(1) 


Seldom  Occasional 

(ri  PI 


u 

w 


Likelihood  of  Occurrence 


Consequence  slightly  outweighs 
Likelihood  (60%  to  40%) 

The  risk  levels  in  this  matrix  are 
consistent  with  FM  5-19 


Tiei 


1)  Gaps  are  first  binned  according  to  severity  (consequence)  and  probability 
(likelihood)  IAW  diagram  above  (Tier  I  =  EH,  H  risk,  Tier  II  =  M  risk,  Tier  111= 
Lrisk) 

□  Ensures  gaps  are  ordered  according  to  highest  risk 

2)  Within  bins,  gaps  scored  according  to: 

□  Scenario  value- based  on  the  consequence  and  likelihood  of  the 
scenario  where  the  gap  derived  from 

□  Task  value  -degree  of  impact  on  the  force  if  the  task  is  not  performed 


Risk  Value-  based  on  COE's  assessment  of  Capability 
Gap’s  risk  to  Organization's  mission 
accomplishment  in  scenarios. 

Tier! -Critical-  Must  Fully  Solve  or  Mitigate  to  Ensure 
Mission  Accomplishment 

Tier  II  -  Moderate  -  Develop  &  implement  Solution  Strategies  as 
Tier  I  is  resolved 

Tier  III  -  Low-  Plan  mitigation  &  solve  as  times  resources 

allow 


Figure  10.  Overall  Operational  Risk  Categories 

The  Overall  Risk  assessed  served  as  a  bin  to  segregate  gaps  qualitatively  before  the  gap  value  model 
detennined  preference  among  gaps  within  the  bins. 

Computation  of  a  gap’s  aggregated  risk  value  across  tasks,  formations  and  scenarios  can  be  defined  as: 

GOO  =  ZLiZ?=iZk=isifijkJk[gijk(yijk')]  (3) 

Thus,  a  comparison  of  G(Y)  for  each  gap  and  its  characterized  risk  in  accordance  with  ATP  5-19 
(Extremely  High  Operational,  High  Operational  Risk,  Moderate  Operational  Risk,  and  Low  Operational 
Risk)  provides  a  means  to  prioritize  the  gaps  according  to  their  risk  value  of  not  meeting  the  Anny’s 
tasks  across  formations  and  scenarios. 


Values  for  Ideas  for  Non-Materiel  and  Materiel  Approaches  (INMA/IMA) 
Alternatives  to  Solve  Capability  Gap  to  enable  Preference 
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To  solve  a  gap,  experts  assessed  individual  INMA/IMA  ability.  The  application  of  the  additive  value 


model  was  done  in  a  similar  manner,  as  described  above,  in  order  to  enable  investment  in  the  most 


preferred  INMA/IMA  for  the  highest  risk  gaps.  Again,  the  scenarios  i,  formations  j  and  tasks  t  attributes 
remain  the  same,  e.g.,  So,  in  this  instance,  we  consider  an  additional  attribute,  gap  risk,  since 


we  are  concerned  about  solving  highest  risk  gaps  soonest.  Thus,  let  the  weight  associated  with  r  gaps 
that  an  INMA/IMA  helps  solve  or  mitigate  be  defined  as  the  normalized  value  of  gap  risk,  gijkiyijk )•  It 

is  simply  added  to  the  model  and  has  the  value  given  by  gnAyuk)  —  ll>kCl>k —  jt  js  the  normalized 

Lijk=Olikjcijk 

value  for  a  gap’s  risk  associated  with  the  INMA/IMA  solving  or  mitigating  gap  r across  task,  k, 
formation,  j,  and  scenario,  i,  so  that  Yujk=i Qijk  =  1  •  Now  for  a  feasible  mitigating  solution/idea,  (zijkr) 
associated  with  gap  r,  let,  Mijkr{zijkr)  be  the  value  of  the  product  of  the  extent  it  provides  to  mitigate 


the  gap  r  in  task  k  of  formation  j  in  scenario  i  on  a  0-100  scale  and  a  constant,  solution/idea  feasibility, 
k,  as  shown  in  Figure  1 1 .  Thus,  let  A,  be  the  measure  of  feasibility  of  a  solution/idea,  z,  which  is  a 
constant  value  no  matter  which  gap  r  it  is  associated  with,  in  task  k  of  formation  j  and  scenario  i,  so  that, 
A  =  Tr  Sp  Af. 


Extremely  Large  Extent -The  solution/idea  mitigates  all  (>80%) 
identified  deficiencies  in  standards  or  condition  of  the  gap  in  the 
scenano  and  formation  and  reduces  the  gap  multiple  levels  to  an 
acceptable  level  of  nsk  (Low-Moderate) 

Large  Extent  -The  solution/idea  mitigates  the  majority  (50-80%)  of 
the  identified  deficiencies  in  standards  or  condition  of  the  gap  and 
achieves  a  reduction  in  gap  risk  one  or  more  levels  indudingto  an 
acceptable  level  of  nsk  (at  least  Moderate). 

Moderate  Extent-The  solution/idea  mitigates  many  (49-25%)  of 
the  identified  deficiencies  in  standards  or  condition  of  the  gap 
This  level  of  mitigation  may  not  be  sufficient  to  reduce  nsk  to  an 
acceptable  level  but  reduces  nsk  one  or  more  levels 

Slight  Extent-The  solution/idea  mitigates  a  some  (24-10%)  of  the 
identified  deficiencies  in  standards  or  condition  of  the  gap  This 
level  of  mitigation  is  notsufficientto  reduce  gap  nsk  any  level. 

Minimal  Extent-The  solution/idea  mitigates  few  (<10%)  of  the 
indentified  deficiencies  in  standards  or  condition  and  does  not 
mitigate  the  gap  nsk  any  level. 


Figure  11.  Solution  Approach/Idea  Value  Scale  Criteria 
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Then,  the  total  of  values  of  a  solution/idea,  Z,  across  all  gaps  it  helps  mitigate,  is  defined  as  S(Z),  and  is 
given  by: 

S(Z)  =  E?=iE?=iZ2=iE!=iSt  ft]  Ujkdijk  [Mijkr{zijkr)\  (4) 

Also,  a  common  factor  in  all  solution  approaches/ideas  for  solving  or  mitigating  a  capability  gap  is  the 
notion  of  the  feasibility  of  the  solution  achieving  the  mitigation/resolution.  This  factor  is  a  measure  of 
the  uncertainty  of  the  solution/idea  being  embraced  by  the  Army,  investment  made  and  fielding 
achieved.  This  process  often  requires  multiple  years  of  effective  management  to  bring  the  solution  to 
fruition  within  the  Army,  therefore  it  cannot  be  a  foregone  conclusion  that  the  solution  will  exist  in  the 
timeframe  expected  or  required.  Feasibility,  consistent  with  Joint  Capabilities  Integration  and 
Development  System13,  is  a  combination  of  three  factors,  technical  risk  (7>)  associated  with  the 
solution/idea,  supportability  costs  differences  (Sp)  from  similar  approaches  being  used  currently,  and  an 
estimate  of  the  affordability  (/if)  based  upon  similar  approaches  costs  and  takes  the  form 

Thus,  a  comparison  of  S(Z)  across  gaps  provides  a  means  to  prioritize  the  gap  solutions/ideas  according 
to  their  mitigation  value  for  solving  gaps  in  tasks  across  formations  and  scenarios  accounting  for 
uncertainty  in  the  pursuit  of  that  solution/idea.  It  is  important  to  note  that  non-materiel  solutions/ideas 
do  not  have  a  technical  risk  therefore  those  solutions/ideas  were  given  maximum  value  on  the  scale. 

This  insures  consistency  with  the  JCIDS  guidance  to  consider  non-materiel  approaches  prior  to  materiel 
approaches  first  since  they  are  generally  less  cost. 

Given  the  formulation  of  the  CNA  value  models  above,  ARCIC  implemented  a  set  of  web-based  tools, 
and  extensive  training  of  a  more  than  60  subject  matter  expert  assessors  representing  Decision  Makers 
across  the  command  on  the  tools  and  the  metrics  to  ensure  accuracy  of  the  data  collected  associated  with 


13TRADOC  Capabilities-Based  Assessment  (CBA)  Guide,  v3.1,  U.S.  Army  Training  and  Doctrine  Command,  ATFC-OP,  FT 
Eustis,  VA  23604  10  May  2010  p8, 
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the  metrics  using  their  constructed  scales.  It  is  important  to  note  that  no  discussion  of  the  science 
associated  with  the  use  of  scales  and  implementation  of  scales  in  a  survey  (web-based)  of  the  experts  is 
provided  here  as  it  is  complex  and  would  distract  from  the  primary  purpose  of  this  paper.  The  next 
section  discusses  the  results  and  finding  of  the  implementation  of  the  value  model  approach  as  outlined 
here.  Included  is  discussion  of  the  level  of  confidence  the  Anny  Leadership/decision  makers  can  have 
in  the  CNA  recommendations. 

Implementation  Results  Discussion 

In  order  for  the  value  models  to  be  used  in  providing  recommended  preferences  from  the  community  of 
practice  to  decision  makers,  the  community,  as  well  as,  the  decision  makers  must  have  confidence  that 
the  resulting  recommendations  are  the  correct  representation  of  the  community’s  preferences  and  the 
value  measures  represent  the  attribute  being  measured  as  precisely  as  possible.  Thus,  the  results  must 
reflect  the  values  of  the  decision  makers  and  the  community  of  practitioners  in  the  art  and  science  of 
Capabilities  Development  and  be  mathematically  correct  in  their  implementation.  This  is  a  particularly 
challenging  requirement  considering  the  scope  of  the  CNA,  where  there  are  multiple  levels  of 
practitioners  and  decision  makers.  Discussion  on  getting  it  right  could  consume  many  pages  in  this 
paper,  so  only  a  summary  of  the  effort  to  achieve  “right”  is  provided  here.  Drawing  on  the  experience 
of  the  TRADOC  Analysis  Center  (TRAC),  and  the  Center  for  Army  Analysis  (CAA)  and  the  wisdom  of 
senior  leaders  and  analysts  an  understanding  of  what  right  looks  like  was  achieved.  Essentially,  the  goal 
for  the  CNA  analysis  was  to  achieve  a  “good  enough’  ’  standard  which  included  the  extent  of  functions 
across  the  Anny  to  be  assessed,  the  number  of  detailed/integrated  versus  segmented  fonnation 
assessments,  the  extent  of  the  representation  of  the  current  and  programmed  capabilities,  and  the  number 
and  breadth  of  situations  (scenarios)  to  represent  the  realm  of  possible  operational  environments  and 
threats.  Early  in  the  initiation  of  the  CNA,  these  were  presented  to  the  community  and  the  decision 
makers  for  resolution.  The  decision  makers  made  adjustments  given  the  resources  available,  but  in  the 
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end  the  community  wholeheartedly  supported  the  resulting  preferences  as  representing  the  Anny’s 
ability  to  meet  its  Joint  Warfighting  requirements,  even  though  “holes”  were  identified.  Thus,  the  level 
of  “good  enough”  acceptability  of  the  results  was  achieved  up  to  and  including  the  4-star  level.  From  a 
scientific  perspective,  however,  spurred  on  by  a  peer  review  conducted  by  CAA,  detailed  post-hoc 
analysis  showed  that  adjustments  to  some  processes,  attributes,  and  metrics  are  warranted  to  achieve 
greater  precision  in  the  preference  recommendations  and  acceptability.  Accordingly,  the  post-hoc 
analysis  included:  an  analysis  to  examine  value  function  relevance  including  analysis  of  significant 
differences  in  means  derived  from  sampling  of  the  capabilities  development  population  where  sampling 
of  experts  was  used;  sensitivity  analysis  of  changes  in  attribute  weightings  effect  on  preferences;  and 
correlation  sensitivities  between  ranks  of  weighted  attributes  and  alternative  ranks. 

Measure  Value  Function  Relevance  and  Testing  for  Measure  Sample  Mean  Differences 

Accurate  assessment  of  measures  for  an  attribute  in  the  value  model  is  essential  to  determination  of 
preference  among  alternatives,  especially  when  the  attribute  is  to  be  used  as  a  key  weight  in  the  value 
model.  In  a  perfect  model  of  the  type  needed  here  (where  few  if  any  natural  scales  exist  thus 
quantitative  analysis  is  not  wholly  feasible),  it  is  desirable  to  have  a  qualitative  value  function  with  a 
constructed  scale  with  non-overlapping  scale  increments  with  well-defined  criteria  associated  with 
each  scale  increment  in  order  to  use  the  entire  extent  of  the  scale  increment  vice  grouping  preferences 
in  only  one  or  two  of  the  scale  increments.14  Additionally,  to  have  confidence  in  the  selected 
preferences  among  alternative,  it  is  important  that  there  is  consensus  in  the  value  samples  and 
significant  differences  in  the  means  exist  where  sampling  was  used  to  determine  values  on  the  scale 
that  will  be  used  for  the  weights  in  the  value  model.  To  ensure  preference  determination  is  not 

14  Ewing,  L.,  Dell,  R.F.,  MacCalman,  M.,  Whitney,  L.,  Capability  Portfolio  Analysis  Tool  (CPAT)  Verification  and  Validation 
Report.  Defense  Technical  Information  Center,  NPS-OR-13-OOl,  Naval  Postgraduate  School,  Monterey  California,  January 
2013. 
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artificially  constrained  by  the  scale  of  the  metric  selected  to  measure  an  attribute,  analysis  alternatives 
should  be  measured  over  a  large  part  of  the  range  of  the  scale,  (e.g.,  distributed  across  the  scale  vice 
focused  in  only  one  or  two  areas  of  the  scale).  Generally,  the  relevance  of  the  measure  to  make 
preference  selections  would  be  questionable  should  assessors:  value  all  alternative  in  only  a  small 
section  of  the  possible  scale;  there  was  a  lack  of  consensus  for  sampled  values;  or  there  was  no 
differences  in  the  means  for  the  sampled  values. 


Scenario  Risk  to  Mission  Accomplishment  Attribute.  The  scenario  risk  attribute  was  one  of  two 
attributes  where  a  panel  of  experts  from  across  the  Army  was  surveyed  to  determine  the  preference  for 
the  alternatives.  For  the  scenario  attribute,  examination  of  Figure  12  shows  the  plotted  range  of 
scenario  sample  means  on  the  constructed  scales  used  as  a  swing  weight  in  the  value  model  for 
‘Likelihood  of  Occurrence  of  Anny  Operations’  measure  and  the  scenario  means  in  the  value  model 
for  ‘Consequence  to  the  force  mission  accomplishment’  measure  in  the  scenario  risk  attribute.  For 
likelihood  of  occurrence  measure,  the  means  of  assessor’s  characterization  of  all  five  scenarios  were  in 
the  occasional  level  of  occurrence  using  9%  of  the  total  possible  scale. 


Scenario  Likelihood  of  Occurrence 
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FIGURE  12.  Scenario  Value  Function  Relevance 
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The  result  was  predictable  in  that  these  five  scenarios  were  prescribed  by  the  Department  of  Defense 
as  likely  to  occur.  However,  the  result  of  small  variation  in  the  means  is  more  attributed  to  of  a  lack  of 
consensus  among  the  assessors  as  to  exactly  how  likely  the  Army  would  be  to  face  operations  similar 
to  those  in  the  provided  scenarios.  Examination  of  Figure  13  shows  the  lack  of  consensus  result  even 
though  two  iterations  of  Delphi  method  were  used  to  try  to  achieve  consensus.  A  one-way  ANOVA 
with  a  95%  confidence  level  for  the  likelihood  measure  showed  no  significant  difference  among  the 
means  (p>.05).  Thus,  the  likelihood  measure  for  the  scenario  attribute  should  be  considered 
ineffective  in  assisting  in  detennining  preference  of  one  scenario  as  more  likely  than  another. 
Accordingly,  the  likelihood  measure  was  only  used  to  establish  a  degree  of  likelihood  of  each  scenario 
as  one  component  in  the  Scenario  Operational  Risk  measure  and  no  statement  of  the  likelihood  of 
occurrence  of  the  Army  performing  operations  similar  to  the  operations  presented  in  the  scenarios  is 
made  as  part  of  the  overall  result. 


Figure  13.  Likelihood  of  Occurrence  Value  Function  Data 

For  the  consequence  measure,  the  means  were  in  two  of  four  possible  consequence  categories,  Critical 
and  Marginal  consequence  spanning  41%  of  the  total  possible  scale.  This  indicates  that  the 
consequence  scale  was  marginally  relevant  in  distinguishing  among  scenario  consequence  preferences. 
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Examination  of  Figure  14  reveals  that  there  was  a  good  level  of  consensus  among  assessors  about  the 
degree  of  consequence  to  the  Army’s  mission  accomplishment  should  operations  be  required  in  a 
scenario.  A  one-way  ANOVA  with  95%  confidence  level  for  the  mean  consequence  values  revealed 
significant  differences  among  the  means  (p<.05).  As  can  be  seen  by  inspection  of  Figure  12,  moderate 
consensus  in  the  sample  exists  and  a  Tukey  Method  test  revealed  groupings  of  similar  consequence 
means  between  EPP,  NEA  and  ISC-B;  between  ISC  -B  and  SSSP;  and  Between  SSSP  and  AFR. 


Figure  14.  Consequence  Value  Function  Data 

Thus,  the  consequence  measure  serves  adequately  to  distinguish  preference  among  the  scenarios.  The 
result  for  the  scenario  risk  to  the  Army  attribute  (product  of  likelihood  and  consequence)  was 
dominated  by  the  consequence  preferences  thus  the  derived  scenario  weights  also  had  significant 
differences  sufficient  to  enable  distinguishing  between  the  risks  of  scenarios. 


Formation  Contribution  Attribute 
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The  formation  attribute  was  the  second  of  two  attribute  measures  using  a  sampling  of  the  Army’s 
Formation  Design  experts.  Inspection  of  Figure  15  reveals  a  wide  distribution  of  formation  means  for 
the  ‘Extent  of  Contribution  to  Scenario  Mission  Accomplishment’  measure  from  the  constructed  scales 
used  as  a  swing  weight  in  the  value  model.  The  result  indicates  that  the  80%  of  the  scale  was  used  in 
each  scenario  to  differentiate  between  formations’  contribution  to  mission  accomplishment  in  that 
scenario.  However,  no  formation  was  assessed  in  the  extremely  large  extent  criteria  in  any  scenario. 


Formation  Assessment  Criteria  Histograms 


AfftlO  EPP  SC-6  NEA  SSSP 

Scenario 

■  irte-it  •  L»ten*  *  Moderate  fa  tent  Latent  •txtremely  Lirjfe  Latent 


Figure  15.  Distribution  of  Formation  Extent  of  Contribution  measure  Means  by  Scenario 


Inspection  of  the  criteria  and  definition  (see  Figure  7)  reveals  that  it  was  not  possible  to  assess  any 
alternative  (formation)  in  that  criterion  simply  because  no  one  formation  performs  all  tasks  or  functions 
required  in  any  one  of  the  scenarios.  A  minor  correction  to  the  definition  for  the  criteria  that  provides  the 
next  increment  above  large  extent  could  possibly  resolve  this  error  and  should  be  implemented  in  the 
next  iteration  of  CNA.  A  one-way  ANOVA  of  the  formation  means  in  each  scenario  revealed 
differences  in  each  of  the  5  scenarios  (p<.05).  Figure  16,  contains  the  result  of  a  Tukey  method  test  of 
means  in  each  of  the  5  scenarios  to  determine  where  there  were  significant  differences  between 
formation  means  in  each  scenario  with  90%  simultaneous  confidence  interval. 
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Figure  16.  Results  of  Tukey  Method  Test  of  Formation  Measure  Sample  Means 

Groupings  are  shown  by  common  letter  associations  and  similar  color  cells.  The  Tukey  method  test 
showed  that  there  are  statistically  significant  differences  in  the  means  of  the  formations  in  each  scenario. 
The  preference  of  formations  (as  well  as  sets  of  formations  where  means  are  not  different)  that  best 
contribute  to  mission  accomplishment  in  each  scenario  vary.  Also,  by  inspection  of  Figure  15,  there  are 
large  groupings  where  the  means  are  the  same  for  multiple  formation  alternatives.  This  result  was 
anticipated  from  post  sample  interviews  of  assessors  that  revealed  most  assessors  felt  that  groups  of 
formations  were  equally  critical  for  conducting  operations  in  the  scenario.  In  fact,  most  operations, 
including  the  scenarios  presented  in  the  analysis,  are  sourced  with  multiple  formations  performing 
different  functions,  which  as  a  whole,  represent  the  capability  needed  to  achieve  mission 
accomplishment  across  the  all  6  phases  of  an  operation.  Recall  that  the  measure  was  designed  to 
capture  the  value  of  the  formation’s  function  in  the  scenario  without  distinction  of  the  phase  of 
operations.  Thus,  it  was  possible  for  an  assessor  to  equally  value  a  formation  that  primarily  perfonned 
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its  function  in  phase  II  of  operations  with  one  that  primarily  perfonned  its  function  in  phase  III.  This 
distinction  was  not  made  during  the  survey,  thus  assessors  made  assumptions  about  a  formations 
contribution  based  upon  knowledge  of  operational  phases  that  weren’t  being  measured,  leading  to 
equalities  among  large  groups  of  formations  in  the  scenarios.  Thus,  providing  clarity  on  formation  value 
by  phase  of  operation  in  comparison  to  one  another  would  likely  enable  greater  distinction  among 
formations.  Additionally,  providing  another  meaningful  criterion  upon  which  to  make  distinctions 
between  formation  contributions  to  scenario  objective  achievement  would  improve  this  attributes 
contribution  to  alternative  preference.  Despite  this  weakness  in  distinguishing  among  individual 
formation  contribution  to  scenario  mission  accomplishment,  and  the  addition  of  another  attribute 
criterion,  the  attribute’s  contribution  to  alternative  preference  is  sufficient  to  engender  confidence  in  the 
value  model  to  produce  correct  results. 

Task  Contribution  to  Formation  Mission  Accomplishment  Attribute. 

The  Task  attribute  was  the  first  attribute  in  the  value  model  not  derived  from  sampling  of  experts  but  by 
eliciting  value  from  a  consolidated  judgment  of  the  proponent  (experts  and  leaders)  responsible  for  the 
task  perfonnance  in  the  Anny.  Hence,  the  attribute’s  and  measures’  relevance  was  assessed  only  by 
analysis  of  the  distribution  of  alternative  among  the  measure  criteria.  (No  evaluation  of  significant 
differences  between  means  is  possible  since  the  consensus  was  achieved  within  a  meeting  of  experts  at 
the  proponent).  Inspection  of  Figure  17  generally  reveals  a  distribution  of  Task  alternatives  across  the 
measure  criterion  that  enables  distinction  of  fonnation  contribution  where  few  fonnations  where  highly 
valued,  most  formations  were  moderately  valued  and  few  fonnations  were  not  valued.  The  alternative 
assessments  on  the  constructed  scales  achieved  an  average  across  scenarios  of  98  and  97.5%  of  the 
possible  scale  respectively  for  degree  of  fonnations  mission  accomplishment  and  impact  on  scenario 
Force’s  mission.  One  trend  was  noted.  The  exception  to  the  expected  distribution  across  criterion  is 


33 


Unclassified 


ARCIC-TR-015-01 
1  OCT  2015 

shown  in  Figure  17  in  both  measures  of  a  task’s  contribution  to  formation  mission  accomplishment  in 
the  two  scenarios  where  the  task  apportionment  was  skewed  to  the  top  two  criteria.  An  explanation  for 
this  came  from  post-survey  interviews  with  the  proponents  who  nearly  consistently  stated  that  the 


Formation's  Task  Assessment  Criteria  Histogram 
Impact  on  Scenario's  Force  Mission  Accomplishment 


Formation's  Task  Assessment  Criteria  Histogram 
Degree  of  Formation  Mission  Accomplishment 


Figure  17.  Distribution  of  Task  Alternative  across  Measure  Criterion 


increased  risk  and  urgency  associated  with  combat  operations  of  those  scenarios  caused  a  corresponding 


increase  in  the  tasks  value  performed  by  formations.  Simply,  it  was  felt  that  more  tasks  were  critical  in 


S1E  and  S3  scenarios.  As  shown  in  Figure  18,  a  comparison  of 
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Figure  18.  Magnitude  of  Difference  in  Two  Task  Criteria 


the  two  criteria,  degree  of  formations  mission  accomplishment  and  impact  on  scenario  Force’s  mission, 


to  one  another  in  each  of  the  scenarios  shows  that  there  was  little  difference  in  the  assessment  scores 
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between  them.  The  distributions  of  difference  in  scores  between  the  two  criteria  are  left  skewed  to  a 
small  difference  with  the  mean  of  the  difference  in  scores  ranging  from  4.4  to  6.4.  Thus,  though  the  use 
of  the  levels  of  each  criterion  is  effective  in  determining  preference  both  criteria  are  nearly  identical 
across  the  levels  for  each  task  in  each  scenario  suggesting  that  both  criteria  are  not  necessary. 
Investigating  the  change  in  solution  value  or  order  by  eliminating  one  of  the  two  criteria  suggests,  as 
shown  in  Figure  19,  that  eliminating  either  criteria  has  little  effect  on  the  solution  values  or  solution 
order. 


Figure  19.  Comparison  of  Task  Assessment  Criteria  Selection  on  Solution  Order  and  Score 


DOTMLPF  Fielded  or  Programmed  Solutions  Achievement  of  the  Task  Attribute 

The  achievement  of  the  task  attribute,  as  pointed  out  earlier,  was  measured  on  a  constructed  scale  of 
extent  of  achievement  of  the  task  with  5  possible  levels  to  return  a  value  for  each  solution.  Inspection  of 
Figure  20  shows  that  solutions  were  distributed  across  the  99%  of  the  scale  (1-100).  However,  for  most 
Formations  or  Warfighting  Functions,  the  distribution  of  DOTMLPF  solutions  is  skewed  toward  the 
lower  end  of  the  scale  indicating  most  solutions  have  smaller  contribution  to  achieving  the  tasks. 

Overall,  57%  of  the  solutions  were  scored  in  the  lower  two  criteria  (minimal  and  slight)  with  only  17% 
in  the  upper  two  criteria  (large  and  extremely  large  extent  of  achievement).  This  was  expected  since 
most  tasks  require  multiple  solutions  each  contributing  some  ability  to  reach  full  achievement  under  the 
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conditions  (across  scenarios)  to  the  listed  condition  or  standards.  Thus,  in  general,  the  measure  was 
adequate  and  relevant  in  assisting  in  distinguishing  between  alternatives’  ability  to  achieve  the  task. 


Distribution  of  DOTMLPF  Solutions 
Across  Task  Achievement  Criteria 


Task  Achievement  Criteria 


Overall  Oistr  ibution  of  Solution*  in  T«*k 
Achievement  Measure 


Figure  20.  Distribution  of  DOTMLPF  Solutions  across  Task  Achievement  Criteria 


Capability  Gap  Operational  Risk  Attributes 

Recall  that  the  Capability  Gap  Operational  Risk  attribute  consists  of  two  measures,  likelihood  that  the 
gap  will  occur  and  consequence  on  force  mission  accomplishment  in  the  scenario  when  the  gap  occurs. 
Examination  of  Figure  21  shows  that  Gap  alternatives  were  assessed  across  99%  of  the  1-100  in  each 
measure.  For  likelihood  of  occurrence  measure  the  gap  alternatives  were  concentrated  in  the  Likely 
(42%)  and  Occasional  (47%)  criteria.  For  consequence  measure  the  gaps  were  concentrated  in  the 
critical  (45%)  and  marginal  criteria  (50%).  Both  of  these  results  are  easily  explainable  given  that  there 
are  few  tasks  for  which  the  Army  cannot  develop  a  work-around  using  doctrine  or  rapid  changes  to  force 
structure. 
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Distribution  of  Capability  Gaps 

Across  Consequence  Measure  Criteria 
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Figure  21.  Distribution  of  Capability  Gaps  across  Risk  Criteria 


Thus,  the  both  measures  for  the  Gap  risk  attribute  were  relevant  in  distinguishing  preference  among  the 
identified  capability  gaps. 


Idea  for  Non-materiel  or  Materiel  Solution  Approach  Extent  of  Gap  Mitigation  Attribute 

The  attribute  associated  with  preference  between  Solution  Approach/Idea  is  the  extent  the  Solution/Idea 
contributes  to  mitigating  a  capability  gap.  As  shown  in  Figure  22,  the  solutions/ideas  were  assessed 
across  the  attribute  criteria  with  a  preponderance  of  the  solutions  rated  in  the  middle  categories  and  a 
small  number  in  the  upper  and  lower  criteria  categories.  This  indicates  that  the  criterion  is  adequate  and 
sufficient  to  enable  preference  among  the  solutions/ideas. 
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Distribution  of  Solution  Approaches/Ideas  Across  Gap  Mitigation  Criterion 


Figure  22.  Distribution  of  Gap  Solution  Approaches/Ideas  across  Gap  Mitigation  Criteria 


Alternative  Preference  Sensitivity  to  Changes  in  Attribute  Weightings 

Although  the  preceding  analysis  is  necessary  for  understanding  of  the  relevance  of  the  measures 
contribution  to  preference,  the  more  conventional  sensitivity  analysis  of  an  additive  value  model  is  to 
examine  the  sensitivity  of  changes  in  alternative  preference  to  changes  in  the  weights  of  the  attributes 
comprising  the  value  hierarchy15.  The  simplest  (least  computationally  intensive)  and  most  common 
sensitivity  analysis  technique  is  the  one-at-a-time  analysis  where  one  weight  is  varied  from  0-1  while 
maintaining  the  others  constant.16  Evaluating  the  results  of  the  changes  in  alternative  preference  is  best 
conceptualized  by  graphing  alternative  preference  compared  to  attribute  weight  change  and  noting, 
especially  in  the  relevant  measured  range  of  the  attribute,  the  extent  of  intersection  of  the  lines 

15  Kirkwood,  G.W.  Strategic  Decision  Making;  Multiobjective  Decision  Analysis  with  Spreadsheets,  Wadsworth,  1997,  p82. 

16  Hamby,  D.M.  1994.  A  Review  of  Techniques  for  Parameter  Sensitivity  Analysis  of  Environmental  Models.  Environ. 
Monit.  and  Assess,  ,pl39. 
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representing  the  alternatives.  Intersecting  lines  (non-equal  linear  slopes)  indicate  preference  change  as 
the  attribute  changes  weighted  value.  The  more  alternative  lines  intersect,  the  greater  the  preference  for 
the  alternative  is  sensitive  to  the  attribute  weight.  Ideally,  it  is  desired  to  have  little  or  no  sensitivity  to 
change  in  attribute  weights.  Thus,  it  becomes  important  to  closely  examine  an  attribute  for  unintended 
effects  (e.g.,  errors)  where  the  alternative’s  preference  is  overly  sensitive  to  changes  in  the  attribute  in 
order  for  decision  makers  to  have  confidence  in  the  model’s  recommended  preferences.  Since  each 
attribute  had  some  irregularities  identified  in  the  attribute  relevance  analysis,  we  examine  the 
alternatives  preference  sensitivity  to  each  of  the  four  swing  weights  (scenario  risk  to  mission 
accomplishment,  fonnation  contribution  to  mission  accomplishment,  task  contribution  to  fonnation 
mission  accomplishment,  and  for  gap  solutions,  the  gap’s  risk  to  the  Anny)  that  are  part  of  the  3  additive 
value  models  using  the  one-at-a-time  analysis  and  the  Sensitivity  Index  (SI)17  of  the  alternatives  for  each 
attribute.  We  also,  then  where  appropriate  use  statistical  techniques  to  gain  further  understanding  of  the 
one-at-a-time  analysis  and  SI  results. 

Sensitivity  of  DOTMLPF-P  Solutions  to  Sustain  in  the  POM  Preference  to  Scenario  Weight 

Figures  23  show  the  sensitivity  of  the  preferential  order  of  DOTMLPF  solutions  given  a  change  in 
scenario  weights.  Since  there  are  more  than  1400  solutions  alternatives  we  examine  alternatives  at 
the  top  of  the  order  (where  programs  must  be  sustained  in  the  POM  to  avoid  high  risk),  and  where 
decision  makers  arbitrarily  chose  to  delineate  between  groups  of  solutions  (i.e.,  Tier  I  and  Tier  II,  and, 
Tier  II  and  Tier  III).  Inspection  of  the  Figure  23  shows  that  the  preference  for  the  top  20  solutions  are 
insensitive  to  changes  in  scenario  weights  in  the  relevant  range  (of  the  20  examined  only  two,  solution 


17  Hamby,  D.M.  1994.  A  Review  of  Techniques  for  Parameter  Sensitivity  Analysis  of  Environmental  Models.  Environ. 
Monit.  and  Assess,  pi 40. 
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19  and  17  trade  a  single  order).  Thus,  we  have  high  confidence  in  the  value  model  distinguishing 
preference  in  the  top  solutions  using  the  scenario  attribute. 


Solution  Preference  Change  as  Scenario  Weight  Varies 

Top  20  Solutions 


0  95 
09 

q  0  B5 

3 

4J  08 
C  0.75 

o 

3  0.7 

O 

&  0  65 


08 


0.55 


0.5 


9999999999999999999999999999999999999999999 

UiUiujujuiiiiUiUiUJiiiUiUiUiUiUiiMiuuiuiUiUiuiaiiuuiUiUJUituaJtuuiuiiifuiujUiUJUiaiiuuiUJ 

- - 


•  N  M  N  fl  «  «  n  ' 


Scenario  Weight  Relevant  Range 


Figure  23.  Top  20  DOTMLPF  Solution  Preference  Sensitivity  to  Scenario  Weight  Variation 


Examination  of  Figure  24  shows  significant  sensitivity  of  the  solutions  alternatives  surrounding  the  Tier 
delineations  indicating  that  with  a  change  in  scenario  weight  the  solutions  would  easily  change  tier. 


Figure  24.  Solution  Preference  Sensitivity  to  Scenario  Weight  Variation  at  Decision  Tier  Interface 
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The  Tier  II/III  boundary  is  of  particular  concern,  as  a  great  many  of  the  solutions  would  change  tiers 
with  slight  change  in  scenario  weight.  This  result  is  at  least  partly  due  to  the  focused  attention  to 
assessments  in  tier  I  and  decreasing  attention  to  assessments  outside  of  tier  I  observed  during  the 
elicitation  process.  Thus,  solution  value  changes  would  result  in  solutions  on  both  sides  of  these 
arbitrary  interfaces  switching  tiers  making  the  tier  boundaries  ineffective  in  distinguishing  preferences, 
and  thus  not  aide  decision  making.  This  suggests  that  for  future  assessments,  to  improve  acceptability 
of  the  results,  an  analysis  of  the  sensitivities  of  solution  values  to  scenario  weight  changes  could  be  used 
to  set  the  tier  boundaries  by  minimizing  preference  changes  across  potential  boundaries  in  lieu  of 
arbitrary  establishment  of  those  boundaries  along  quintiles. 

DOTMLPF-P  Solutions  Attribute  Preference  Sensitivities  due  to  Formation  Weight  Variation. 

The  sensitivity  of  solution  preference  to  variation  in  weights  used  for  the  formations  contribution  to 
scenario  mission  achievement  attribute  in  the  mutli-attribute  value  model  is  shown  in  Figure  25.  By 
inspection  of  Figure  25(a),  solution  preference  for  the  top  20  solutions  shows  that  there  is  some 
sensitivity  to  variation  of  the  formation  weights  across  the  relevant  range  (elicited  range  of  weights  for 
each  formation).  Analysis  of  the  change  shows  that  the  change  in  preference  varies  only  a  few  positions 
at  most  across  the  range.  Inspection  of  Figure  25  (b)  and  (C)  shows  the  alternative  sensitivity  at  the  tier 
boundaries,  tier  I/II  and  tier  II/III  respectively.  Again,  the  sensitivity  was  of  greater  concern  at  the  tier 
boundaries  since  this  implies  that  solutions  in  one  tier  would  cross  tier  boundaries  for  changes  across 
formation  weight  range.  Thus,  like  the  scenario  attribute,  except  for  at  the  tier  boundary,  the  solution 
alternative  preference  sensitivity  to  fonnation  weight  change  is  minimal  and  acceptable  to  give 
confidence  in  the  preference  of  the  alternative. 
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Figure  25.  DOTMLPF  Solution  Preference  Sensitivity  to  Formation  Weight  Variation 


Although  not  ideal,  this  degree  of  sensitivity  was  deemed  acceptable  for  the  purposes  of  the  CNA. 
However,  taking  action  to  improve  the  use  of  the  fonnation  criteria  for  the  attribute  and  selecting  the 
proper  tier  boundary  using  the  one-at-a-time  analysis  is  necessary  to  increase  confidence  in  the  tiering 
technique. 
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Solution  Preference  Sensitivities  due  to  Task  Weight  Variation 

The  third  set  of  attributes  weighting  used  in  solution  preference  is  a  Tasks  weight.  It  consisted  of  2 
separate  attributes  that  were  multiplied  together  and  normalized  for  the  set  of  tasks  in  each  formation  in 
each  scenario.  Figure  26  shows  the  sensitivity  of  a  solution’s  value  given  variation  of  task  weight  from 
0-1  while  maintaining  the  other  weights  proportionately  constant.  Figure  25(a)  shows  that  that  there  is 
sensitivity  to  task  weight  change  among  the  top  20  alternatives  examined  and  (b)  and  (c)  also  show 
sensitivity  to  variation  of  task  weight  of  solutions  at  the  arbitrary  tiering  boundaries.  In  these  three 
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Figure  26.  Solution  Alternative  Preference  Sensitivity  Due  to  Task  Weight  Variation 


instances  this  sensitivity  to  tasks  may  reduce  confidence  in  solution  preference  if  it  was  the  result  of 
either  type  1  or  2  errors  in  the  constructed  scale  and  criterion,  or  the  elicitation  process.  Prior  analysis  of 
the  two  separate  attribute  criterion  determined  the  assessors  used  the  entire  scale  range;  however, 
assessors  treated  both  attributes  for  the  task  equally.  Thus,  a  second  criterion  did  not  help  in  the 
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preference.  Revision  of  the  second  criterion  is  required  to  help  with  preference.  This  may  also  help 
reduce  the  solution  alternative  sensitivity  to  the  task. 


Gap  Alternative  Preference  Sensitivities  due  to  Scenario,  Formation  and  Task  Weight  Variation 

Similar  to  the  Solution  alternative  sensitivity  analysis,  Figure  27  illustrates  that  the  top  20  (tier  I) 
capability  gap  alternatives  were  not  sensitive  to  scenario  (a)  and  formation  (b)  attribute  weights,  but 
were  sensitive  to  task  (c)  attribute  weights.  Also,  similar  results  to  the  solution  alternative  analysis  at 
the  tier  boundaries  was  observed  in  the  data  for  capability  gaps  between  tier  I  and  II  and  II  and  III. 


Preference  Change  of  Top  20  Gaps 
as  Scenario  Weight  Varies 


Preference  Change  for  Top  20  Gaps 
as  Formation  Weight  Varies 
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Figure  27.  Gap  Preference  Sensitivity  One-to-one  Analysis  (Top  20  Gaps) 
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Gap  Idea  for  Non-Material  and  Materiel  Solution  Approach  Preference  Sensitivities  due  to 
Attribute  Weight 

Similar  to  the  Solution  and  Gap  alternatives  sensitivity  analysis,  Figure  28  illustrates  that  the  top  Ideas 
for  Non-Materiel  and  Ideas  for  Materiel  Approaches  (subsequently  referred  to  as  Ideas)  alternatives 
were  not  sensitive  to  scenario  and  formation  attribute  weights,  but  were  sensitive  to  task  attribute 
weights. 
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Figure  28.  INMA/IMA  Preference  Sensitivity  One-to-one  Analysis  (Top  20  Ideas) 
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Confirming  the  One-to-One  Sensitivity  Analysis 

Hamby  suggests18  that  a  quantitative  measure  of  alternative’s  preference  sensitivity  to  attributes  weights 
can  be  measured  effectively  by  a  Sensitivity  index.  The  index  (SI)  is  given  by: 

Dmax  —  Dm  in 

SI  = - 

Dmax 

Where  the  Dmax  is  the  maximum  value  for  the  alternative  when  the  attribute  weight  is  at  maximum 
variation  and  Dmin  is  the  minimum  value  of  the  alternative  when  the  attribute  weight  is  at  minimum 
variation.  Solutions  with  indices  close  to  1  are  sensitive  and  those  with  indices  near  0  are  insensitive  to 
the  attribute  weight.  Applying  the  Sensitivity  Index  to  the  CNA  solution  attribute  across  more  than 
1400  solution  alternatives  resulted  in  histograms  of  number  of  solutions  in  each  SI  bin  as  shown  in 
Figure  29.  Inspection  of  the  figure  shows  a  distribution  of  solution  alternative  SI  near  zero  for  the 
scenario  and  fonnation  attribute  weights,  while  the  distribution  of  solution  alternative  SI  for  task 
attribute  weights  is  spread  across  the  indices  scale  with  a  concentration  near  1 .  This  data  supports  the 
one-to-one-analysis  of  the  solution  alternative  sensitivity.  For  the  scenario  and  formation  attribute 
weights,  the  solution  preference  is  relatively  insensitive  compared  to  the  task  attribute  weight.  As  with 
the  one-to-one  analysis  the  solution  preference  is  sensitive  to  the  variation  in  task  weight. 


18  Hamby,  D.M.  1994.  A  Review  of  Techniques  for  Parameter  Sensitivity  Analysis  of  Environmental  Models.  Environ. 
Monit.  and  Assess.,  pi 40. 
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SI  Histogram 
For  Variation  in 
Scenario  Weight 


SI  Histogram 
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Figure  29.  Histograms  of  Solution  Attribute  Sensitivity  Index 


Other  Observed  Relationships  between  Alternatives  and  Attributes 

To  better  understand  the  sensitivity  of  the  alternative’s  preference  to  the  attribute  weight,  we  examined 
the  relationship  between  the  two  variables  (attribute  weight  and  alternative  preference)  using  statistical 
correlation  tests.  Since  a  test  of  normality  and  linearity  showed  the  data  associated  with  attribute  and 
alternatives  was  not  linear,  non-normal,  but  monotonic,  the  Spearman  Rank  Coefficient  was  chosen  to 
analyze  the  variable’s  relationship.  The  Spearman’s  Rank  Correlation  coefficient  (p)  is  highlighted  as 
one  of  the  more  straight  forward  non-parametric  methods  to  use  in  additive  value  model  sensitivity 
analysis19.  Specifically,  the  correlation  between  the  rank  of  the  alternatives  and  the  paired  rank  of  the 


19  Hamby,  D.M.  1994.  A  Review  of  Techniques  for  Parameter  Sensitivity  Analysis  of  Environmental  Models.  Environ. 
Monit.  and  Assess.,  pi 40, 
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number  formations  or  tasks  associated  with  the  alternatives.  After  transforming  the  number  of 
formations  and  tasks  paired  to  each  attribute  (solution  or  gap)  to  ranks  across  the  Speannan  Rank 
Coefficient  for  was  computed.  Table  2  shows  the  correlation  between  ranks  of  the  alternative  (across 
the  row)  and  ranks  of  the  attribute  (down  the  column).  Thus,  there  is  no  significant  correlation  between 
the  fonnation  and  solution  ranks  or  between  the  solution  and  task  number  ranks. 


Paired  Alternatives  Rank  Compared 

Spearman’s  Rank  Coefficient  (p) 

Set  of  All 
Alternatives 

Set  of 

M  Alternatives 
Only 

Set  of  M 
Programmed 
Alternatives  Only 

Solution 

Alternatives 

Formation  Rank 

(based  on  number  of  formations 
where  solution  existed) 

.380 

TBP 

.361 

Task  Rank 

(based  on  number  of  tasks 
solution  helped  achieve) 

.647 

.734 

.754 

INMA/IMA 

Alternatives 

Formation  Rank 

(based  on  number  of  formations 
where  INMA/IMA  existed) 

0.018 

.056 

NA 

Task  Rank 

(based  on  number  of  tasks 
iNMA/IMA  helped  achieve) 

0.623 

.659 

NA 

Gap 

Alternatives 

Formation  Rank 

(based  on  number  of  formations 
where  gaps  existed) 

0.615 

NA 

NA 

Task  Rank 

(based  on  number  of  tasks  gaps 
helped  achieve) 

0.466 

NA 

NA 

Table  2.  Spearman’s  Rank  Coefficient  for  Paired  Alternative  Rank  and  Attribute  Rank 
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Summary  and  Recommendations  for  Continued  Refinement 

Compensatory  Additive  Value  modeling,  a  widely  accepted  Decision  Analysis  technique,  was  applied  to 
assist  in  resource  decisions  on  numerous  alternatives  across  multiple  conflicting  objectives.  The  scope 
of  the  application  involved  three  separate  but  related  value  models  to  determine  preference  on  more  than 
1400  DOTMLPF-P  solution  alternatives,  800  Capability  Gap  alternatives,  and  300  gap  solution 
alternatives.  This  scope  made  implementation  complex,  requiring  extensive  resources  (manpower 
effort,  and  technical  solutions),  and  collaboration  among  more  than  20  stakeholder  agencies.  An 
automated  infonnation  system  was  developed  to  collect,  assess  and  analyze  data  which  enable  the 
complex  collaboration  required  to  achieve  success.  None  the  less,  those  complexities  clearly  introduced 
some  error  in  the  results.  However,  the  results  were  widely  accepted  as  the  best  analytically  robust 
available  to  enable  resource  decision  making  by  the  community  and  stands  as  the  command  position  for 
use  by  all.  Detailed  analysis  of  the  model  elements  compared  with  the  intended  purpose  of  the  element 
revealed  areas  where  refinement  could  improve  confidence  in  the  results.  Analysis  of  the  use  of  the 
attribute  scales  and  criteria  showed  most  of  these  model  elements  were  effective  in  contributing  to 
preference.  However,  the  scenario  attribute  likelihood  criteria,  and  formation  criteria  should  be  refined 
to  allow  for  measurement  across  the  entire  attribute’s  scale  for  better  preference  detennination. 
Sensitivity  analysis  of  attribute  weights  for  the  each  of  the  3  value  models  showed  preference  was  not 
significantly  sensitive  to  3  of  four  attributes  used.  The  preference  for  alternatives  in  each  of  the  three 
values  models  was  sensitive  to  the  weights  of  the  attribute  associated  with  the  contribution  of  tasks  to 
mission  accomplishment  and  requires  refinement  to  improve  preference  confidence.  This  improvement 
includes  eliminating  one  of  the  two  criteria  since  the  data  showed  that  assessors  did  not  or  could  not 
distinguish  between  the  criteria.  Adding  a  distinct  criteria  may  aid  in  preference  detennination  and 
reduce  the  strong  sensitivity  to  this  attribute  that  was  observed.  Additionally,  a  linkage  between  model 
1  and  3  to  enable  comparison  of  the  values  of  programs  and  emerging  idea’s  that  could  become 
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programs  is  desired  by  many  of  the  stakeholders.  This  would  be  relatively  straight  forward  since  both 
models  share  the  same  elements  in  the  value  model.  A  simple  method  to  relate  their  value  to  task  for 
programs  and  the  value  to  gap  solution  would  be  required.  This  would  logically  be  possible  as  a  gap  is 
the  unachieved  portion  of  a  task  (or  multiple  tasks). 
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Acronym 

Expansion 

AAR 

After-action  Report 

ACF 

Anny  Concept  Framework 

ACoE 

Aviation  Center  of  Excellence 

ACSIM 

Assistant  Chief  of  Staff  for  Installation  Management 

AEWE 

Anny  Expeditionary  Warrior  Experiment 

AMC 

Anny  Materiel  Command 

AMEDD 

Anny  Medical  Department 

AOC 

Anny  Operating  Concept 

ArCAT 

Anny  Capabilities  Analysis  Tool 

ARCIC 

Anny  Capabilities  Integration  Center 

ARSTAF 

Anny  Staff 

ASA(ALT) 

Assistant  Secretary  of  the  Anny  for  Acquisition,  Logistics  and  Technology 

ASCC 

Anny  Service  Component  Command 

AWFC 

Anny  Warfighting  Challenges 

AWG 

Asymmetric  Warfare  Group 

AUTL 

Anny  Universal  Task  List 

BDE 

Brigade 

BL 

Battle  Lab 

BLCSE 

Battle  Lab  Collaborate  Simulation  Environment 

CAB 

Combat  Aviation  Brigade 

CAC 

Combined  Arms  Command 

CARD 

Capabilities  and  RAM  Division 

CASCOM 

Combined  Anns  Support  Command 

CBA 

Capabilities-based  Assessment 

52 


Unclassified  ARCIC-TR-015-01 

1  OCT  2015 


Acronym 

Expansion 

CCoE 

Cyber  Center  of  Excellence 

CDD 

Capabilities  Development  Directorate 

CDID 

Capability  Development  Integration  Directorate 

CG 

Commanding  General 

CNA 

Capabilities  Needs  Analysis 

CoC 

Council  of  Colonels 

CoE 

Center  of  Excellence 

CoL 

Campaign  of  Learning 

COIN 

Counterinsurgency 

CPR 

Capability  Portfolio  Review 

CSA 

Combat  Support  Agency 

CWMD 

Counter  Weapons  of  Mass  Destruction 

Cyber  STE 

Cyber  Set  the  Experiment 

DoD 

Department  of  Defense 

DOTD 

Directorate  of  Training  Development  and  Doctrine 

DOTMLPF-P 

Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel,  Facilities  and 
Policy  solutions 

DP 

Decision  Point 

DPG 

Defense  Planning  Guidance 

DTG 

Date-T  ime-Group 

EAD 

Echelons  Above  Division 

EEA 

essential  elements  of  analysis 

EPP 

Enhanced  Protective  Posture 

FAA 

Functional  Area  Analysis 
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Acronym 

Expansion 

FCoE 

Fires  Center  of  Excellence 

FM 

Force  Modernization 

FNA 

Functional  Needs  Analysis 

FORSCOM 

U.S.  Army  Forces  Command 

FSA 

Functional  Solutions  Analysis 

FY 

Fiscal  Year 

GAMEX 

W  argame/Exercise 

GOSC 

General  Officer  Steering  Committee 

HQ  TRADOC 

Headquarters,  U.S.  Anny  Training  and  Doctrine  Command 

HQDA 

Headquarters,  Department  of  Army 

IAW 

In  accordance  with 

I/A/SBCT 

Infantry/ Armor/Stryker  Brigade  Combat  Team 

ICW 

In  Coordination  With 

ICDT 

Integrated  Capabilities  Development  Team 

ICoE 

Intelligence  Center  of  Excellence 

ICW 

In-coordination  with 

IICDT 

Intelligence  Integrated  Capabilities  Development  Team 

IMA 

Ideas  for  Material  Approaches 

INMA 

Ideas  for  Non-material  Approaches 

JAG 

Judge  Advocate  General 

JCIDS 

Joint  Capabilities  and  Integration  Development  System 

JIEDDO 

Joint  Improvised  Explosive  Device  Defeat  Organization 

JTC 

Joint  Training  Center 

IMA 

Idea  for  Materiel  Approach 
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Acronym 

Expansion 

INMA 

Idea  for  Non-Materiel  Approach 

ISR 

Intelligence,  Surveillance,  Reconnaissance 

IWfF 

Intelligence  Warfighting  Function 

LIRA 

Long-range  Investment  Requirement  Analysis 

MSEL 

Mission  Essential  Task  List 

MCCoE 

Mission  Command  Center  of  Excellence 

MCoE 

Maneuver  Center  of  Excellence 

MSCoE 

Maneuver  Support  Center  of  Excellence 

MSO 

Major  Subordinate  Organization 

NIE 

Network  Integration  Evaluation 

NLT 

No  later  than 

PACOM 

Pacific  Command 

PBR 

Program  Budget  Review 

PED 

Processing,  Exploitation,  &  Dissemination 

PEG 

Program  Evaluation  Group 

PIR 

Priority  Intelligence  /  Infonnation  Request 

PMJ 

Professional  Military  Judgment 

POC 

Point  of  Contact 

POM 

Program  Objective  Memorandum 

PPBE 

Planning,  Programming,  Budgeting  and  Execution 

RC 

Required  Capability 

REF 

Rapid  Equipping  Force 

RSVP 

Request  for  Response 

S&T 

Science  and  Technology 
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Acronym 

Expansion 

SCoE 

Sustainment  Center  of  Excellence 

SIMEX 

Simulation/Exercise 

SMDC 

Space  and  Missile  Defense  Command 

SSA 

Support  to  Strategic  Analysis 

SSI 

Soldier  Support  Institute 

STE  WG 

Set  the  Environment  Working  Group 

SWG 

Seminar  Wargame 

SME 

Subject  Matter  Expertise/Subject  Matter  Expert 

SWG 

Seminar  War  Game 

T/C/S 

Tasks,  Conditions,  &  Standards 

TOE 

Table  of  Organization  and  Equipment 

TRAC 

TRADOC  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TRISA 

TRADOC  Intelligence  Support  Activity 

TWV 

Tactical  Wheeled  Vehicle 

UC 

Unified  Challenge 

UJLT 

Universal  Joint  Task  List 

ULO 

Unified  Land  Operations 

UQ 

Unified  Quest 

USASOC 

U.S.  Army  Special  Operations  Command 

WfF 

Warfighting  Function 
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Capability — The  ability  to  complete  a  task  or  execute  a  course  of  action  under  specified  conditions  and 
level  of  perfonnance. 

Capability  developer  (CAPDEV)  --A  person  who  is  involved  in  analyzing,  determining,  prioritizing,  and 
documenting  requirements  for  doctrine,  organizations,  training,  leader  development  and  education,  materiel 
and  materiel-centric  DOTMLPF  requirements,  personnel,  facilities  and  policy  implications  within  the 
context  of  the  force  development  process.  Also  responsible  for  representing  the  end  user  during  the  full 
development  and  lifecycle  process  and  ensures  all  enabling  capabilities  are  known,  affordable,  budgeted,  and 
aligned  for  synchronous  fielding  and  support. 

Capability  Gap — The  inability  to  meet  or  exceed  a  capability  requirement,  resulting  in  an  associated 
operational  risk  until  closed  or  mitigated.  The  gap  may  be  the  result  of  no  fielded  capability,  lack  of 
proficiency  or  sufficiency  in  a  fielded  capability  solution,  or  the  need  to  replace  a  fielded  capability 
solution  to  prevent  a  future  gap. 

Capability  Need — See  “Capability  Requirement.” 

Capability  Requirement — A  capability  required  to  meet  an  organization’s  roles,  functions,  and  missions 
in  current  or  future  operations.  To  the  greatest  extent  possible,  capability  requirements  are  described  in 
relation  to  tasks,  standards,  and  conditions  in  accordance  with  the  Universal  Joint  Task  List  or 
equivalent  DoD  Component  Task  List.  If  a  capability  requirement  is  not  satisfied  by  a  capability 
solution,  then  there  is  also  an  associated  capability  gap.  A  requirement  is  considered  to  be  “draft”  or 
“proposed”  until  validated  by  the  appropriate  authority. 

Capability  Requirement  Document — any  document  used  to  articulate  either  deliberate  or 
urgent/emergent  capability  requirements  and  associated  infonnation  pertinent  to  review  and  validation. 
Capability  Solution — A  materiel  solution  or  non-materiel  solution  to  satisfy  one  or  more  capability 
requirements  and  reduce  or  eliminate  one  or  more  capability  gaps. 
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Capstone  concept  —A  capstone  concept  is  a  holistic  future  concept  that  is  a  primary  reference  for  all 
other  concept  development.  This  overarching  concept  provides  direct  linkages  to  national  and  defense 
level  planning  documents.  A  capstone  concept  drives  the  development  of  subordinate  concepts.  For 
example,  the  CCJO  drives  development  of  joint  concepts  and  service  concepts.  TP  525-3-0  drives  the 
development  of  the  Army  operating  and  functional  concepts. 

Contingency  Operation — A  military  operation  that  (a)  is  designated  by  the 
Secretary  of  Defense  as  an  operation  in  which  members  of  the  Anned  Forces  are  or  may  become 
involved  in  military  actions,  operations,  or  hostilities  against  an  enemy  of  the  United  States  or  against  an 
opposing  military  force;  or(b)  Results  in  the  call  or  order  to,  or  retention  on,  active  duty  of  members  of 
theUnifonned  Services  under  section  688,  12301(a),  12302,  12304,  12304a, 

12305,  or  12406  of  [title  10],  chapter  15  of  [title  10],  section  712  of  title  14,  or  any  other  provision  of 
law  during  a  war  or  during  a  national  emergency  declared  by  the  President  or  Congress.  (Source:  10 
U.S.C.  101.) 

Community  of  practice  (CoP)--This  is  a  group  of  organizations  with  a  common  interest  in  a  subject  area 
who  interact  to  share  information,  processes,  and  products.  A  CoP  is  defined  by  three  characteristics:  the 
shared  domain  of  interest,  the  relationships  defining  the  community  (typically  networked,  consisting  of  the 
organizations  as  nodes),  and  a  shared  set  of  practices  for  the  subject  area. 

Concept--  A  notion  or  statement  of  an  idea  -  an  expression  of  how  something  might  be  done  -  that  can 
lead  to  an  accepted  procedure  (CJCSI  3010.02C).  A  military  concept  is  the  description  of  methods 
(ways)  for  employing  specific  military  attributes  and  capabilities  (means)  in  the  achievement  of  stated 
objectives  (ends).  An  Army  concept  describes  a  problem  or  series  of  problems  to  be  solved, 
assumptions,  the  future  operational  environment,  the  central  idea,  the  components  of  the  solution,  the 
interaction  of  those  components  in  solving  the  problem,  and  the  required  capabilities  necessary  to 
achieve  desired  effects  and  objectives. 
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Concept  of  operations  (CONOPS)  —A  verbal  or  graphic  statement,  in  broad  outline,  of  a  commander’s 
assumptions  or  intent  regarding  an  operation  or  series  of  operations  (CJCSI  3170.01H).  A  concepts- 
based  CONOPS  is  a  statement,  in  broad  outline,  of  a  commander’s  assumptions  or  intent  about  an 
operation  or  series  of  operations.  It  is  designed  to  give  an  overall  picture  and  a  useful  visualization  of 
how  a  future  operation  would  be  conducted  (TP  71-20-3). 

Constraints,  limitations,  and  assumptions  (CLA)~  Constraints,  limitations,  and  assumptions  provide 
the  framework  for  both  the  study  team  and  the  study  sponsor  to  understand  the  conditions  under  which  a 
study’s  results  are  applicable.  Although  commonly  misrepresented  or  used  interchangeably,  these  three 
terms  are  distinctly  different  in  meaning  and  use  in  the  context  of  a  study.  Constraints,  limitations,  and 
assumptions  bound  (scope)  a  study  effort  by  identifying  what  must  (or  must  not)  and  can  (or  cannot)  be 
accomplished;  frame  the  study  space  and  set  the  stage  for  the  study  team’s  methodology  development; 
serve  as  a  "contract"  between  the  study  sponsor  and  the  study  team;  and  provide  a  basis  for  the  sponsor 
to  reconcile  the  study  results  with  how  the  study  was  done. 

Core  Mission  Area — DoD  core  mission  areas  identified  under  the  most  recent  Quadrennial  Roles  and 
Missions  review  are:  Homeland  Defense  and  Civil  Support  (HD/CS);  Deterrence  Operations;  Major 
Combat  Operations  (MCOs);  Irregular  Warfare;  Military  Support  to  Stabilization  Security,  Transition, 
and  Reconstruction  Operations;  and  Military  Contribution  to  Cooperative  Security. 

(Source:  2009  Quadrennial  Roles  and  Missions  Review  Report.) 

Data  —  A  representation  of  facts,  concepts,  or  instructions  in  a  fonnalized  manner  suitable  for 
communication,  interpretation,  or  processing  by  humans  or  by  automatic  means  (as  it  pertains  to 
modeling  and  simulation). 

Delphi  Method  --  a  widely  used  and  accepted  method  for  gathering  data  from  respondents  within  their 
domain  of  expertise.  The  technique  is  designed  as  a  group  communication  process  which  aims  to 
achieve  a  convergence  of  opinion  on  a  specific  real-world  issue. 
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Document  Sponsor — The  organization  submitting  a  capability  requirement  document.  Solution 
sponsors  for  successor  documents — Capability  Development  Documents  (CDDs),  Capability  Production 
Documents  (CPDs),  and  Joint  DOTMLPF-P  Change  Recommendations  (Joint  DCRs) — may  be  different 
than  the  Requirement  Sponsors  for  initial  documents — Initial  Capabilities  Documents  (ICDs),  urgent 
operational  needs  (UONs),  joint  UONs  (JUONs),  and  joint  emergent  operational  needs  (JEONs). 
Different  Sponsors  for  requirements  and  solutions  can  occur  when  the  initial  document  Sponsor  does  not 
have  acquisition  authority  and  a  different  organization  is  designated  to  develop  and  field  a  capability 
solution,  or  when  one  Sponsor  elects  to  leverage  a  previously  validated  document  generated  by  a 
different  Sponsor. 

DoD  Components — OSD,  the  Military  Departments,  the  Chainnan  of  the  Joint  Chiefs  of  Staff,  the 
CCMDs,  the  Office  of  the  Inspector  General  of  the  Department  of  Defense,  the  Department  of  Defense 
Agencies,  field  activities,  and  all  other  organizational  entities  in  the  Department  of  Defense. 
Experimentation  —  The  exploration  of  innovative  methods  of  operating,  especially  to  assess  their 
feasibility,  evaluate  their  utility,  or  detennine  their  limits  to  reduce  risk  in  the  current  force  (today's 
operations)  and  the  future  force  (developments).  Experimentation  identifies  and  verifies  acceptable 
solutions  for  required  changes  in  DOTMLPF  to  achieve  significant  advances  in  current  and  future 
capabilities.  Experiments  aid  in  validating  the  feasibility  of  future  requirements  determination  efforts. 
TRAC's  Definitions  for  Analysts,  TRAC-TD-05-010  dated  May  2005  defines  experimentation  as:  The 
use  of  an  event  or  series  of  events  designed  to  investigate  concepts  or  prototypes. 

Force  modernization  proponent—  The  HQDA  principal  official,  commander,  commandant,  director,  or 
chief  of  the  respective  center,  school,  institution,  or  agency  with  primary  duties  and  responsibilities 
relative  to  DOTMLPF  and  related  requirements  for  a  particular  function  (i.e.  Combined  Arms  Center  is 
a  force  modernization  proponent,  but  not  a  Center  of  Excellence).  See  AR  5-22. 
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Functional  area  —A  functional  area  is  a  broad  scope  of  related  joint  warfighting  skills  and  attributes 
that  may  span  the  range  of  military  operations.  Specific  skill  groupings  that  make  up  the  functional  areas 
are  approved  by  the  JROC.  See  CJCSI  3170. 

Gap — See  “Capability  Gap.” 

Integrated  Priority  List — A  list  of  a  Combatant  Commander’s  highest  priority  requirements, 
prioritized  across  Service  and  functional  lines,  defining  shortfalls  in  key  programs  that,  in  the  judgment 
of  the  Combatant  Commander,  adversely  affect  the  capability  of  the  Combatant  Commander’s  forces  to 
accomplish  their  assigned  mission.  Also  called  IPL. 

Integrated  capabilities  development  team  (ICDT)  --An  integrated  team  of  key  stakeholders  and 
SMEs  from  multiple  disciplines  chartered  by  Dir,  ARCIC  to  initiate  the  JCIDS  process  through  conduct 
of  the  CBA  to  identify  capability  gaps  in  a  functional  area,  identify  nonmateriel  and/or  materiel 
approaches  to  resolve  or  mitigate  those  gaps,  and  develop  an  ICD  and/or  a  DCR  or  DICR,  when 
directed. 

Joint — Connotes  activities,  operations,  organizations,  etc.,  in  which  elements  of  two  or  more  Military 
Departments  participate.  (JP  1-02.  Source:  JP  1.) 

Joint  Emergent  Operational  Need  (JEON) — UONs  that  are  identified  by  a 

CCMD,  CJCS,  or  VCJCS  as  inherently  joint  and  impacting  an  anticipated  contingency  operation. 

Joint  Military  Requirement — A  capability  necessary  to  fulfill  or  prevent  a  gap  in  a  core  mission  area 
of  the  Department  of  Defense.  (Source:  10  U.S.C.  181.) 

Joint  Urgent  Operational  Need  (JUON) — UONs  that  are  identified  by  a  CCMD, 

CJCS,  or  VCJCS  as  inherently  joint  and  impacting  an  ongoing  contingency  operation. 

Materiel  (Capability  Solution) — All  items  (including  ships,  tanks,  self-propelled  weapons,  aircraft, 
etc.,  and  related  spares,  repair  parts,  and  support  equipment,  but  excluding  real  property,  installations, 
and  utilities)  necessary  to  equip,  operate,  maintain,  and  support  military  activities  without  distinction  as 
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to  its  application  for  administrative  or  combat  purposes.  See  also  equipment;  personal  property.  (JP  1- 
02.  Source:  JP  4-0.) 

Need — See  “Capability  Requirement.” 

Non-Materiel  (Capability  Solution) — Changes  to  doctrine,  organization,  training,  (previously  fielded) 
materiel,  leadership  and  education,  personnel,  facilities,  and/or  policy,  implemented  to  satisfy  one  or 
more  capability  requirements  (or  needs)  and  reduce  or  eliminate  one  or  more  capability  gaps,  without 
the  need  to  develop  or  purchase  new  materiel  capability  solutions.  (Upon  approval  of  this  document,  this 
tenn  and  definition  are  proposed  for  addition  to  JP  1-02.) 

Rapid  Acquisition — A  streamlined  and  tightly  integrated  iterative  approach,  acting  upon  validated 
urgent  or  emergent  capability  requirements,  to:  conduct  analysis  and  evaluate  alternatives  and  identify 
preferred  solutions;  develop  and  approve  acquisition  documents;  contract  using  all  available  statutory 
and  regulatory  authorities  and  waivers  and  deviations  of  such,  appropriate  to  the  situation;  identify  and 
minimize  technical  development,  integration,  and  manufacturing  risks;  and  rapidly  produce  and  deliver 
required  capabilities. 

Requirement — See  “Capability  Requirement.” 

Requirement  Sponsor — See  “Document  Sponsor.” 

Scenario—This  is  a  graphic  and  narrative  description  of  area,  environment,  means  (political,  economic, 
social,  and  military),  and  events  of  a  future  hypothetical  conflict.  Scenarios  provide  a  framework  for 
assessing  the  U.S.  force  capabilities  under  specified  situations;  identifying  potential  improvements  to 
Army,  joint,  and  other  service  DOTMLPF;  and  evaluating  proposed  concepts  and  changes  to  the  Army. 
See  TR  71-4. 

Solution — See  “Capability  Solution.” 

Solution  Sponsor — See  “Document  Sponsor.” 

Sponsor — See  “Document  Sponsor.” 
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Threat — The  sum  of  the  potential  strengths,  capabilities,  and  strategic  objectives  of  any  adversary  that 
can  limit  or  negate  mission  accomplishment  or  reduce  force,  system,  or  equipment  effectiveness.  It  does 
not  include  (a)  natural  or  environmental  factors  affecting  the  ability  or  the  system  to  function  or  support 
mission  accomplishment;  (b)  mechanical  or  component  failure  affecting  mission  accomplishment  unless 
caused  by  adversary  action;  or  (c)  program  issues  related  to  budgeting,  restructuring,  or  cancellation  of  a 
program. 

Urgent  Operational  Need  (UON) — Capability  requirements  identified  as  impacting  an  ongoing  or 
anticipated  contingency  operation.  If  left  unfulfilled,  UONs  result  in  capability  gaps  potentially  resulting 
in  loss  of  life  or  critical  mission  failure.  When  validated  by  a  single  DoD  Component,  these  are  known 
as  DoD  Component  UONs.  DoD  Components,  in  their  own  tenninology,  may  use  a  different  name  for  a 
UON. 

Validation — The  review  and  approval  of  capability  requirement  documents  by  a  designated  validation 
authority.  The  JROC  is  the  ultimate  validation  authority  for  capability  requirements  unless  otherwise 
delegated  to  a  subordinate  board  or  to  a  designated  validation  authority  in  a  Service,  CCMD,  or  other 
DoD  Component. 
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